Re: [Python-Dev] __file__ and bytecode-only
On Mar 16, 2010, at 07:44 PM, Nick Coghlan wrote: In Python 3.1, *invoking* py_compile.compile() will create 2.x style bytecode. Similarly, when force==False, compileall.compile_dir() and compileall.compile_path() will check for 2.x style bytecode in order to decide whether or not to compile the module. The question for 3.2 is what bytecode layout py_compile.compile() should generate. For the precompile-a-system-library use case it should clearly generate a PEP 3147 layout and this probably makes sense as the default behaviour in 3.2. However, for production of bytecode-only packages, it would be convenient to be able to explicitly invoke the 2.x style behaviour without having to specify the target filename explicitly using the 'cfile' parameter (which isn't exposed at the compileall layer anyway). My working branch modifies py_compile to produce PEP 3147 layout by default. I agree that it should support traditional pyc output as an option, and compileall should support this as well. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Barry Warsaw wrote: On Mar 14, 2010, at 12:17 AM, Nick Coghlan wrote: While it's probably OK if the import side-effects only create files using the new scheme, the standard library modules will likely need to support both schemes (although I'm not sure if same as import system or same as Python 3.1 make more sense as the default semantics - probably the former). I don't understand this point. compileall probably /could/ be extended to understand bytecode-only (i.e. legacy or 3.2) layout. I've added that to the PEP too. In Python 3.1, *invoking* py_compile.compile() will create 2.x style bytecode. Similarly, when force==False, compileall.compile_dir() and compileall.compile_path() will check for 2.x style bytecode in order to decide whether or not to compile the module. The question for 3.2 is what bytecode layout py_compile.compile() should generate. For the precompile-a-system-library use case it should clearly generate a PEP 3147 layout and this probably makes sense as the default behaviour in 3.2. However, for production of bytecode-only packages, it would be convenient to be able to explicitly invoke the 2.x style behaviour without having to specify the target filename explicitly using the 'cfile' parameter (which isn't exposed at the compileall layer anyway). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Mar 14, 2010, at 12:17 AM, Nick Coghlan wrote: Hmm - methinks the PEP actually needs to talk explicitly about the py_compile and compileall modules. These compile the files directly rather than using the import system's side-effect, so they'll need to understand the intricacies of the new system. Good point. I'll add this to the PEP. While it's probably OK if the import side-effects only create files using the new scheme, the standard library modules will likely need to support both schemes (although I'm not sure if same as import system or same as Python 3.1 make more sense as the default semantics - probably the former). I don't understand this point. compileall probably /could/ be extended to understand bytecode-only (i.e. legacy or 3.2) layout. I've added that to the PEP too. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Mar 14, 2010, at 04:37 PM, Paul Moore wrote: The bdist_wininst installer also compiles modules explicitly on install (as does the python.org Windows MSI installer). I've always assumed that this worked via compileall, but haven't checked. Regardless, these should probably also be covered in the PEP. Thanks; added to PEP. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Mar 15, 2010, at 07:43 AM, Nick Coghlan wrote: He did (in favour of keeping the directory visible). http://www.mail-archive.com/python-dev@python.org/msg45203.html (added to pep) -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Fri, Mar 12, 2010 at 05:56:57PM -0500, Barry Warsaw wrote: Nope, sorry I should have been clearer. *creation* is the key here. As per BDFL pronouncement, we'll support reading pyc-only modules just like we do today. This is in PEP 3147. We won't support creating them though. BTW, I'm actually starting to work on the implementation now, so if you want to play along: % bzr branch lp:~barry/python/pep3147 Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a hidden directory? ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Sun, 14 Mar 2010 11:59:50 pm Jon Ribbens wrote: Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a hidden directory? I disagree with your assumption that there is *any* use-case for a hidden directory, let alone an ideal one. I despise hidden directories and dot files. I know it is the Unix way, and I suppose it made sense back in 1975 when users had two or three dot files in their home directory, but I count 215 dot files in my home directory compared to only 77 visible files, and I have no idea how most of them got there or what they do. Programs that litter the file system with dot files are bad enough when they do it in $HOME, but sprinkling dot files everywhere they can is inexcusable. This is not the place for me to rant over the evil that is dot files, so I'll just say this: Python works on platforms other than Unix/Linux, and some of those platforms don't treat dot files as anything more than a file with a leading dot in the file name. -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Le Sun, 14 Mar 2010 12:59:50 +, Jon Ribbens jon+python-...@unequivocal.co.uk a écrit : Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a hidden directory? I suppose the same reason pyc files are not hidden nowadays. We want people to be aware that these files exist. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On 13 March 2010 14:17, Nick Coghlan ncogh...@gmail.com wrote: The creation side could be made a little more explicit in the PEP. We could also do something via the compileall module. (Pause while Nick goes and reads the source code for compileall for the first time ever...) Hmm - methinks the PEP actually needs to talk explicitly about the py_compile and compileall modules. These compile the files directly rather than using the import system's side-effect, so they'll need to understand the intricacies of the new system. The bdist_wininst installer also compiles modules explicitly on install (as does the python.org Windows MSI installer). I've always assumed that this worked via compileall, but haven't checked. Regardless, these should probably also be covered in the PEP. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Sun, 14 Mar 2010 12:59:50 -, Jon Ribbens jon+python-...@unequivocal.co.uk wrote: On Fri, Mar 12, 2010 at 05:56:57PM -0500, Barry Warsaw wrote: Nope, sorry I should have been clearer. *creation* is the key here. As per BDFL pronouncement, we'll support reading pyc-only modules just like we do today. This is in PEP 3147. We won't support creating them though. BTW, I'm actually starting to work on the implementation now, so if you want to play along: % bzr branch lp:~barry/python/pep3147 Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a hidden directory? You did miss it. Perhaps a link to the discussion should be added to the PEP, or a summary of it. If I remember correctly the BDFL even weighed in on this point. -- R. David Murray www.bitdance.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
R. David Murray wrote: On Sun, 14 Mar 2010 12:59:50 -, Jon Ribbens jon+python-...@unequivocal.co.uk wrote: Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a hidden directory? You did miss it. Perhaps a link to the discussion should be added to the PEP, or a summary of it. If I remember correctly the BDFL even weighed in on this point. He did (in favour of keeping the directory visible). Here's one reference (although not the main one): http://osdir.com/ml/python-dev/2010-02/msg00114.html My Google-fu is failing me, so I don't have a link to the other part of the thread that I refer to in that message. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Jon Ribbens wrote: Sorry if I missed it, but why on earth is the bytecode directory __pycache__ and not .pycache? (Or indeed anything else that starts with a '.') Surely this is a classic ideal use case for a hidden directory? Please don't try to hide it. On MacOSX, it makes it impossible to look inside using the Finder if you want to, and it won't have any effect on Windows anyway. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Barry Warsaw wrote: *creation* is the key here. As per BDFL pronouncement, we'll support reading pyc-only modules just like we do today. This is in PEP 3147. We won't support creating them though. The creation side could be made a little more explicit in the PEP. We could also do something via the compileall module. (Pause while Nick goes and reads the source code for compileall for the first time ever...) Hmm - methinks the PEP actually needs to talk explicitly about the py_compile and compileall modules. These compile the files directly rather than using the import system's side-effect, so they'll need to understand the intricacies of the new system. While it's probably OK if the import side-effects only create files using the new scheme, the standard library modules will likely need to support both schemes (although I'm not sure if same as import system or same as Python 3.1 make more sense as the default semantics - probably the former). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote: The remaining open question to my mind is whether or not there should be a -X option to control the bytecode generation. E.g.: -Xcache_bytecode=no (don't write bytecode files at all) -B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use case. -Xcache_bytecode=file (write a classic foo.pyc file) -Xcache_bytecode=dir (write to the __pycache__ directory) With cache_bytecode=dir being the default for future releases. I believe we've decided /not/ to support creation of bytecode-only distributions out of the box. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On 12/03/2010 19:53, Barry Warsaw wrote: On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote: The remaining open question to my mind is whether or not there should be a -X option to control the bytecode generation. E.g.: -Xcache_bytecode=no (don't write bytecode files at all) -B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use case. -Xcache_bytecode=file (write a classic foo.pyc file) -Xcache_bytecode=dir (write to the __pycache__ directory) With cache_bytecode=dir being the default for future releases. I believe we've decided /not/ to support creation of bytecode-only distributions out of the box. I thought Guido said on this topic [1]: FWIW, I started at -1 and am still -1. I think the PEP is overreaching in this aspect; it does not serve the stated purpose of the PEP to make life easier for distros that share code between Python versions. Has something changed since then? Michael [1] http://mail.python.org/pipermail/python-dev/2010-March/098098.html -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (BOGUS AGREEMENTS) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On 12/03/2010 22:48, Michael Foord wrote: On 12/03/2010 19:53, Barry Warsaw wrote: On Mar 04, 2010, at 11:34 PM, Nick Coghlan wrote: The remaining open question to my mind is whether or not there should be a -X option to control the bytecode generation. E.g.: -Xcache_bytecode=no (don't write bytecode files at all) -B and $PYTHONDONTWRITEBYTECODE will still be supported and serve this use case. -Xcache_bytecode=file (write a classic foo.pyc file) -Xcache_bytecode=dir (write to the __pycache__ directory) With cache_bytecode=dir being the default for future releases. I believe we've decided /not/ to support creation of bytecode-only distributions out of the box. I thought Guido said on this topic [1]: Ok - so oops, I misunderstood. I'll go back into my corner now. :-) All the best, Michael FWIW, I started at -1 and am still -1. I think the PEP is overreaching in this aspect; it does not serve the stated purpose of the PEP to make life easier for distros that share code between Python versions. Has something changed since then? Michael [1] http://mail.python.org/pipermail/python-dev/2010-March/098098.html -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (BOGUS AGREEMENTS) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (BOGUS AGREEMENTS) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Mar 12, 2010, at 10:48 PM, Michael Foord wrote: On 12/03/2010 19:53, Barry Warsaw wrote: I believe we've decided /not/ to support creation of bytecode-only distributions out of the box. I thought Guido said on this topic [1]: FWIW, I started at -1 and am still -1. I think the PEP is overreaching in this aspect; it does not serve the stated purpose of the PEP to make life easier for distros that share code between Python versions. Has something changed since then? Nope, sorry I should have been clearer. *creation* is the key here. As per BDFL pronouncement, we'll support reading pyc-only modules just like we do today. This is in PEP 3147. We won't support creating them though. BTW, I'm actually starting to work on the implementation now, so if you want to play along: % bzr branch lp:~barry/python/pep3147 -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Henning von Bargen wrote: If the ZIP contains only bytecode files, it is just not intended for changing any code, so I don't think this is an argument. If you have access to the source code, you still can use that instead of messing around with byte code. That doesn't apply when it is the app developer trying to figure out why their app is breaking when they throw it in a bytecode only zip :) It's a moot point anyway - Guido has pronounced that the bytecode-only support will be left alone by PEP 3147 even if the other caching changes are eventually accepted. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Barry Warsaw wrote: On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote: I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only? Couldn't vendors just replace the real .py files with empty files? Yes, I think that's a possibility. What would people think about that? I actually thought of this, but didn't post it because it defeats the point of byte-code only distribution. - putting text in the .py file will break the application - touching the .py file in any way will break the application If someone wants to break the bytecode only support it should be its own PEP and not coupled with PEP 3147. The remaining open question to my mind is whether or not there should be a -X option to control the bytecode generation. E.g.: -Xcache_bytecode=no (don't write bytecode files at all) -Xcache_bytecode=file (write a classic foo.pyc file) -Xcache_bytecode=dir (write to the __pycache__ directory) With cache_bytecode=dir being the default for future releases. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Mar 04, 2010, at 11:25 PM, Nick Coghlan wrote: It's a moot point anyway - Guido has pronounced that the bytecode-only support will be left alone by PEP 3147 even if the other caching changes are eventually accepted. Right. I should have stopped while I was ahead. We'll just keep what's in the PEP. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Nick Coghlan wrote: Another option is to remove bytecode-only support from the default filesystem importer, but keep it for zipimport (since the stat call savings don't apply in the latter case). +1 Baptiste Carvello wrote: However, making a difference between zipimport and the filesystem importer means the application will stop working if I unzip the library zip file, which is surprising. Unzipping the zip file can be handy when debugging a bug caused by a forgotten module. If the ZIP contains only bytecode files, it is just not intended for changing any code, so I don't think this is an argument. If you have access to the source code, you still can use that instead of messing around with byte code. Henning ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Mar 02, 2010, at 09:06 PM, Steven D'Aprano wrote: (1) What happens if the __cache__ directory doesn't exist and the enclosing directory is unwriteable, or if it does exist, but is unreadable? I expect that the byte code files will simply not be created, and everything will continue without them. s/__cache__/__pycache__ but yes, just as it does today. (2) Presumably this only effects imports, not running python source code as a script. If I do this: python myscript.py from the shell, I would expect that no __cache__ directory will be created, just like today. Correct. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] __file__ and bytecode-only
I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only? Couldn't vendors just replace the real .py files with empty files? Then no one would need the extra stat call, and no one would be bitten by orphaned .pyc files after a rename. [Yes, zips could still allow unmatched names; yes, it would be helpful if a tool were available to sync the last-modification time; yes a deprecation release should still be needed.] -jJ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote: I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only? Couldn't vendors just replace the real .py files with empty files? Yes, I think that's a possibility. What would people think about that? -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Wed, Mar 3, 2010 at 16:37, Jim Jewett jimjjew...@gmail.com wrote: I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only? Couldn't vendors just replace the real .py files with empty files? Because if someone screws up the mod time on the source files the .pyc files will get recreated silently. -Brett Then no one would need the extra stat call, and no one would be bitten by orphaned .pyc files after a rename. [Yes, zips could still allow unmatched names; yes, it would be helpful if a tool were available to sync the last-modification time; yes a deprecation release should still be needed.] -jJ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On approximately 3/3/2010 5:49 PM, came the following characters from the keyboard of Barry Warsaw: On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote: I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only? Couldn't vendors just replace the real .py files with empty files? Yes, I think that's a possibility. What would people think about that? -Barry That's kooky, but not as kooky as my idea. As mentioned elsewhere, timestamps would have to be treated carefully. In this scenario, the .pyc files would still live in __pycache__ ? Complete with the foo.token.pyc naming ? So one could actually create a fat .zip application that could work with a variety of installed Python versions ??? Hmm... -- Glenn -- http://nevcal.com/ === A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Barry Warsaw wrote: On Mar 03, 2010, at 07:37 PM, Jim Jewett wrote: Couldn't vendors just replace the real .py files with empty files? Yes, I think that's a possibility. What would people think about that? Seems like a perverse thing to have to do to me. Also a bit fragile, since you would have to make sure that the empty .py files were dated older than the .pyc files and stayed that way, lest Python try to recompile them and wipe out your code. You would also have to be careful to build installers that didn't recompile .py files on installation. (Haven't had much experience building installers using distutils, so I'm not sure how much of a problem that would be.) -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Glenn Linderman wrote: In this scenario, the .pyc files would still live in __pycache__ ? Complete with the foo.token.pyc naming ? It might be neater to have a separate cache directory for each bytecode version, named __cache.token__ or some such. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
On Mar 3, 2010, at 10:22 PM, Greg Ewing wrote: Glenn Linderman wrote: In this scenario, the .pyc files would still live in __pycache__ ? Complete with the foo.token.pyc naming ? It might be neater to have a separate cache directory for each bytecode version, named __cache.token__ or some such. Okay, this is probably some pretty silly bikeshedding, but: if we're going to have it be something.something-else, can we please make sure that .something-else is a common extension that means python bytecode cache? It would be good to keep the file-manager and shell operations required to say blow away bytecode cache directories as simple as possible. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Le Wed, 3 Mar 2010 17:51:04 -0800, Brett Cannon br...@python.org a écrit : On Wed, Mar 3, 2010 at 16:37, Jim Jewett jimjjew...@gmail.com wrote: I understand the need to ship without source -- but why does that require supporting .pyc (or .pyo) -only? Couldn't vendors just replace the real .py files with empty files? Because if someone screws up the mod time on the source files the .pyc files will get recreated silently. Unless the .py files arrange to raise a syntax error on compiling. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Le Thu, 04 Mar 2010 16:22:13 +1300, Greg Ewing greg.ew...@canterbury.ac.nz a écrit : Glenn Linderman wrote: In this scenario, the .pyc files would still live in __pycache__ ? Complete with the foo.token.pyc naming ? It might be neater to have a separate cache directory for each bytecode version, named __cache.token__ or some such. Actually, I find it neater to have a single cache directory. It makes for much less clutter, and simpler ignore rules. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Glyph Lefkowitz wrote: if we're going to have it be something.something-else, can we please make sure that .something-else is a common extension that means python bytecode cache? Maybe something like __bytecode-version__.pycache ? -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Antoine Pitrou wrote: Unless the .py files arrange to raise a syntax error on compiling. I guess that prevents a total disaster, but the program is still broken and you have to hunt down the offending files and fix the timestamps -- if it's even evident what the problem is and how to fix it. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Antoine Pitrou wrote: Actually, I find it neater to have a single cache directory. It makes for much less clutter, and simpler ignore rules. Another possibility would be to have a single top-level cache directory with a subdirectory for each version: __bytecode__.pycache/version/filename.pyc I don't think this ought to increase the stat call count, since the interpreter will always be looking for a particular version of bytecode, so it can just stat for the appropriate subdirectory directly. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__ and bytecode-only
Le Thu, 04 Mar 2010 16:39:03 +1300, Greg Ewing greg.ew...@canterbury.ac.nz a écrit : Antoine Pitrou wrote: Unless the .py files arrange to raise a syntax error on compiling. I guess that prevents a total disaster, but the program is still broken and you have to hunt down the offending files and fix the timestamps -- if it's even evident what the problem is and how to fix it. Bytecode-only distributions are supposed to be created by the vendor/distributor. It's up to him/her to get things right. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Tue, 2 Mar 2010 11:41:52 am Barry Warsaw wrote: After PEP 3147 is implemented, and the default, you'll have to byte-compile the files, then find the pycs in the __pycache__ directory, move them up a level and rename them. Then of course remove the .py files. It's not insurmountable of course, I think if we support pyc-only imports, people are rightly going to want us to write and support the tool to create those imports. Surely that's a job for a tiny Python script, or even a shell script? It doesn't sound hard, not from the description given. I imagine there will be recipes on ActiveState quite quickly, and if there isn't, that would be good evidence that demand for the feature is low. -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Tue, 2 Mar 2010 11:59:55 am Barry Warsaw wrote: Thanks everybody for providing great input on this aspect of the PEP. I've updated the open issues section to include a list of the possible resolutions for bytecode-only imports. Unless anybody has more ideas, it might just be time to get a BDFL pronouncement. Please excuse me if these minor points have already been discussed, but I couldn't see them in the PEP. (1) What happens if the __cache__ directory doesn't exist and the enclosing directory is unwriteable, or if it does exist, but is unreadable? I expect that the byte code files will simply not be created, and everything will continue without them. (2) Presumably this only effects imports, not running python source code as a script. If I do this: python myscript.py from the shell, I would expect that no __cache__ directory will be created, just like today. BTW, you have some sort of automated warning in the PEP: System Message: WARNING/2 (pep-3147.txt, line 237) Title underline too short. http://www.python.org/dev/peps/pep-3147/#id47 -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Tue, Mar 2, 2010 at 02:06, Steven D'Aprano st...@pearwood.info wrote: On Tue, 2 Mar 2010 11:59:55 am Barry Warsaw wrote: Thanks everybody for providing great input on this aspect of the PEP. I've updated the open issues section to include a list of the possible resolutions for bytecode-only imports. Unless anybody has more ideas, it might just be time to get a BDFL pronouncement. Please excuse me if these minor points have already been discussed, but I couldn't see them in the PEP. (1) What happens if the __cache__ directory doesn't exist and the enclosing directory is unwriteable, or if it does exist, but is unreadable? I expect that the byte code files will simply not be created, and everything will continue without them. (2) Presumably this only effects imports, not running python source code as a script. If I do this: python myscript.py from the shell, I would expect that no __cache__ directory will be created, just like today. BTW, you have some sort of automated warning in the PEP: System Message: WARNING/2 (pep-3147.txt, line 237) Title underline too short. http://www.python.org/dev/peps/pep-3147/#id47 It's now fixed. Barry forgot to run the Makefile for the PEPs before checking in. Shame! =) -Brett -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Antoine Pitrou a écrit : Le Sun, 28 Feb 2010 21:45:56 +0100, Baptiste Carvello a écrit : bytecode-only in a zip is used by py2exe, cx_freeze and the like, for space reasons. Disabling it would probably hurt them. Source code compresses quite well. I'm not sure it would make much of a difference. I did a quick check on the stdlib: a zip with .py and .pyc is about 80% bigger than one with .pyc only. If you use only the bytecode, this can be seen as waisted space. On the other hand, if you ever need to debug the application, source is very handy... Anyway, I'm a bit worried if bytecode-only is disabled from zipimport without some input from the developpers of py2exe/cx_freeze/etc, as they are big users of it. Cheers, Baptiste ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Brett Cannon wrote: So there are a total of five to six depending on the OS (actually, VMS goes up to eight!) before a search path is considered not to contain a module. The windows list is actually going to be slightly different (dir, pyd, py, pyw, py[co]). It looks for .pyd files rather than either flavour of .so file (we stopped allowing .dll files some time back due to the sqlite3 DLL naming conflict). So I believe it is always 5 stat calls on the major platform - dropping the bytecode filename check saves 20% of them on a miss. I'm not convinced that saving is worth the hassle of incurring a whole pile of subtle backward compatibility problems. It seems better to say Python doesn't create in-place bytecode files anymore, but if you arrange to put them there yourself we'll still read them. I certainly wouldn't support removing the feature without some solid benchmarks to say that it really is going to significantly speed up typical import times for non-trivial modules (given that part of the definition of non-trivial is significant amounts of code to be run when imported, that bar is likely to be a tough one to clear). And thanks to doing this I realized importlib is not stat'ing the directory first which should fail faster than checking for the __init__ files every time. That would explain why we had different ideas as to what the interpreter was doing :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Le Mon, 01 Mar 2010 09:09:09 +0100, Baptiste Carvello baptiste...@free.fr a écrit : I did a quick check on the stdlib: a zip with .py and .pyc is about 80% bigger than one with .pyc only. If you use only the bytecode, this can be seen as waisted space. On the other hand, if you ever need to debug the application, source is very handy... My point is that the wasted size compared to the total bundle size (with interpreter and 3rd party C libs) would probably be small. Anyway, I'm a bit worried if bytecode-only is disabled from zipimport without some input from the developpers of py2exe/cx_freeze/etc, as they are big users of it. Granted. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Nick Coghlan wrote: Michael Foord wrote: Can't it look for a .py file in the source directory first (1st stat)? When it's there check for the .pyc in the cache directory (2nd stat, magic number encoded in filename), if it's not check for .pyc in the source directory (2nd stat + read for magic number check). Or am I missing a subtlety? The problem is doing this little dance for every path on sys.path. To unpack this a little bit for those not quite as familiar with the import system (and to make it clear for my own benefit!): for a top-level module/package, each path on sys.path needs to be eliminated as a possible location before the interpreter can move on to check the next path in the list. So the important number is the number of stat calls on a miss (i.e. when the requested module/package is not present in a directory). Currently, with builtin support for bytecode only files, there are 3 checks (package directory, py source file, pyc/pyo bytecode file) to be made for each path entry. The PEP proposes to reduce that to only two in the case of a miss, by checking for the cached pyc only if the source file is present (there would still be three checks for a hit, but that only happens at most once per module lookup). While the PEP is right in saying that a bytecode-only import hook could be added, I believe it would actually be a little tricky to write one that didn't severely degrade the performance of either normal imports or bytecode-only imports. Keeping it in the core import, but turning it off by default seems much less likely to have unintended performance consequences when it is switched back on. Another option is to remove bytecode-only support from the default filesystem importer, but keep it for zipimport (since the stat call savings don't apply in the latter case). What if ... a bytecode-only mode is triggered by __main__ loading from a bytecode file, otherwise the .py files are needed and are checked to make sure the bytecode files are current. Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Mon, Mar 1, 2010 at 08:30, Ron Adam r...@ronadam.com wrote: Nick Coghlan wrote: Michael Foord wrote: Can't it look for a .py file in the source directory first (1st stat)? When it's there check for the .pyc in the cache directory (2nd stat, magic number encoded in filename), if it's not check for .pyc in the source directory (2nd stat + read for magic number check). Or am I missing a subtlety? The problem is doing this little dance for every path on sys.path. To unpack this a little bit for those not quite as familiar with the import system (and to make it clear for my own benefit!): for a top-level module/package, each path on sys.path needs to be eliminated as a possible location before the interpreter can move on to check the next path in the list. So the important number is the number of stat calls on a miss (i.e. when the requested module/package is not present in a directory). Currently, with builtin support for bytecode only files, there are 3 checks (package directory, py source file, pyc/pyo bytecode file) to be made for each path entry. The PEP proposes to reduce that to only two in the case of a miss, by checking for the cached pyc only if the source file is present (there would still be three checks for a hit, but that only happens at most once per module lookup). While the PEP is right in saying that a bytecode-only import hook could be added, I believe it would actually be a little tricky to write one that didn't severely degrade the performance of either normal imports or bytecode-only imports. Keeping it in the core import, but turning it off by default seems much less likely to have unintended performance consequences when it is switched back on. Another option is to remove bytecode-only support from the default filesystem importer, but keep it for zipimport (since the stat call savings don't apply in the latter case). What if ... a bytecode-only mode is triggered by __main__ loading from a bytecode file, otherwise the .py files are needed and are checked to make sure the bytecode files are current. That's way too magical for my tastes, especially if you mess up and accidentally leave behind a __main__.pyc after moving the __main__.py file. -Brett Ron ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Ron Adam wrote: What if ... a bytecode-only mode is triggered by __main__ loading from a bytecode file, otherwise the .py files are needed and are checked to make sure the bytecode files are current. That would preclude having a bytecode-only library that could be used by a sourceful program. Such a situation might arise if you have an application with a scripting interface that is used by importing stuff from the application's internal libraries. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Feb 28, 2010, at 10:40 PM, Antoine Pitrou wrote: File extensions exist for a reason, even if you find that kooky and have strong ideas about the psychology of text editors. Having some binary files named foobar.py would certainly annoy a lot of people, including me. I completely agree. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Feb 28, 2010, at 02:51 PM, Greg Ewing wrote: A solution might be to look for the presence of the cache directory, and only look for a .pyc in the source directory if there is no cache directory. Testing for the cache directory would only have to be done once per package and the result remembered, so it would add very little overhead. I think the other thing that bothers me about continuing to support pyc-only imports, is that people will then want tools to create them. Right now, it's probably just as easy as byte-compiling everything, then finding the .py files and removing them. After PEP 3147 is implemented, and the default, you'll have to byte-compile the files, then find the pycs in the __pycache__ directory, move them up a level and rename them. Then of course remove the .py files. It's not insurmountable of course, I think if we support pyc-only imports, people are rightly going to want us to write and support the tool to create those imports. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sun, Feb 28, 2010 at 02:51:16PM +1300, Greg Ewing wrote: Floris Bruynooghe wrote: (But even then I'm not convinced that would double the stat calls for normal users, only for those who only ship .pyc files) It would increase the number of stat calls for normal users by 50%. You would need to look for a .pyc in the source directory, then .py in the source directory and .pyc in the cache directory. That's compared to two stat calls currently, for .py and .pyc. Can't it look for a .py file in the source directory first (1st stat)? When it's there check for the .pyc in the cache directory (2nd stat, magic number encoded in filename), if it's not check for .pyc in the source directory (2nd stat + read for magic number check). Or am I missing a subtlety? A solution might be to look for the presence of the cache directory, and only look for a .pyc in the source directory if there is no cache directory. Testing for the cache directory would only have to be done once per package and the result remembered, so it would add very little overhead. That would work too, but I don't understand yet why the .pyc check in the source directory can't be done last. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
-- http://www.ironpythoninaction.com On 28 Feb 2010, at 12:19, Floris Bruynooghe floris.bruynoo...@gmail.com wrote: On Sun, Feb 28, 2010 at 02:51:16PM +1300, Greg Ewing wrote: Floris Bruynooghe wrote: (But even then I'm not convinced that would double the stat calls for normal users, only for those who only ship .pyc files) It would increase the number of stat calls for normal users by 50%. You would need to look for a .pyc in the source directory, then .py in the source directory and .pyc in the cache directory. That's compared to two stat calls currently, for .py and .pyc. Can't it look for a .py file in the source directory first (1st stat)? When it's there check for the .pyc in the cache directory (2nd stat, magic number encoded in filename), if it's not check for .pyc in the source directory (2nd stat + read for magic number check). Or am I missing a subtlety? The problem is doing this little dance for every path on sys.path. Michael A solution might be to look for the presence of the cache directory, and only look for a .pyc in the source directory if there is no cache directory. Testing for the cache directory would only have to be done once per package and the result remembered, so it would add very little overhead. That would work too, but I don't understand yet why the .pyc check in the source directory can't be done last. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Michael Foord wrote: Can't it look for a .py file in the source directory first (1st stat)? When it's there check for the .pyc in the cache directory (2nd stat, magic number encoded in filename), if it's not check for .pyc in the source directory (2nd stat + read for magic number check). Or am I missing a subtlety? The problem is doing this little dance for every path on sys.path. To unpack this a little bit for those not quite as familiar with the import system (and to make it clear for my own benefit!): for a top-level module/package, each path on sys.path needs to be eliminated as a possible location before the interpreter can move on to check the next path in the list. So the important number is the number of stat calls on a miss (i.e. when the requested module/package is not present in a directory). Currently, with builtin support for bytecode only files, there are 3 checks (package directory, py source file, pyc/pyo bytecode file) to be made for each path entry. The PEP proposes to reduce that to only two in the case of a miss, by checking for the cached pyc only if the source file is present (there would still be three checks for a hit, but that only happens at most once per module lookup). While the PEP is right in saying that a bytecode-only import hook could be added, I believe it would actually be a little tricky to write one that didn't severely degrade the performance of either normal imports or bytecode-only imports. Keeping it in the core import, but turning it off by default seems much less likely to have unintended performance consequences when it is switched back on. Another option is to remove bytecode-only support from the default filesystem importer, but keep it for zipimport (since the stat call savings don't apply in the latter case). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sun, Feb 28, 2010 at 11:07:27PM +1000, Nick Coghlan wrote: Michael Foord wrote: Can't it look for a .py file in the source directory first (1st stat)? When it's there check for the .pyc in the cache directory (2nd stat, magic number encoded in filename), if it's not check for .pyc in the source directory (2nd stat + read for magic number check). Or am I missing a subtlety? The problem is doing this little dance for every path on sys.path. To unpack this a little bit for those not quite as familiar with the import system (and to make it clear for my own benefit!): for a top-level module/package, each path on sys.path needs to be eliminated as a possible location before the interpreter can move on to check the next path in the list. Aha, that was the clue I was missing. Thanks! Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sun, Feb 28, 2010 at 05:07, Nick Coghlan ncogh...@gmail.com wrote: Michael Foord wrote: Can't it look for a .py file in the source directory first (1st stat)? When it's there check for the .pyc in the cache directory (2nd stat, magic number encoded in filename), if it's not check for .pyc in the source directory (2nd stat + read for magic number check). Or am I missing a subtlety? The problem is doing this little dance for every path on sys.path. To unpack this a little bit for those not quite as familiar with the import system (and to make it clear for my own benefit!): for a top-level module/package, each path on sys.path needs to be eliminated as a possible location before the interpreter can move on to check the next path in the list. So the important number is the number of stat calls on a miss (i.e. when the requested module/package is not present in a directory). Currently, with builtin support for bytecode only files, there are 3 checks (package directory, py source file, pyc/pyo bytecode file) to be made for each path entry. Actually it's four: name/__init__.py, name/__init__.pyc, name.py, and then name.pyc. And just so people have terminology to go with all of this, this search is what the finder does to say whether it can or cannot handle the requested module. The PEP proposes to reduce that to only two in the case of a miss, by checking for the cached pyc only if the source file is present (there would still be three checks for a hit, but that only happens at most once per module lookup). Just to be explicit, Nick is talking about name/__init__.py and name.py (note the skipping of looking for any .pyc files). At that point only the loader needs to check for the bytecode in the __pycache__ directory. While the PEP is right in saying that a bytecode-only import hook could be added, I believe it would actually be a little tricky to write one that didn't severely degrade the performance of either normal imports or bytecode-only imports. Keeping it in the core import, but turning it off by default seems much less likely to have unintended performance consequences when it is switched back on. It all depends on how it is implemented. If the bytecode-only importer stats a directory to check for the existence of any source in order to decide not to handle it, that is an extra stat call, but that is only once per sys.path/__path__ location by the path hook, not every attempted import. Now if I ever manage to find the time to break up the default importers and expose them then it should be no more then adding the bytecode-only importer to the chained finder that already exists (it essentially chains source and extension modules). Another option is to remove bytecode-only support from the default filesystem importer, but keep it for zipimport (since the stat call savings don't apply in the latter case). That's a very nice option. That would isolate it into a single importer that doesn't impact general performance for everyone else. -Brett Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sun, 2010-02-28 at 12:21 -0800, Brett Cannon wrote: Actually it's four: name/__init__.py, name/__init__.pyc, name.py, and then name.pyc. And just so people have terminology to go with all of this, this search is what the finder does to say whether it can or cannot handle the requested module. Aren't there also: name.so namemodule.so ? -Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Nick Coghlan a écrit : Another option is to remove bytecode-only support from the default filesystem importer, but keep it for zipimport (since the stat call savings don't apply in the latter case). bytecode-only in a zip is used by py2exe, cx_freeze and the like, for space reasons. Disabling it would probably hurt them. However, making a difference between zipimport and the filesystem importer means the application will stop working if I unzip the library zip file, which is surprising. Unzipping the zip file can be handy when debugging a bug caused by a forgotten module. Cheers, Baptiste ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Brett Cannon wrote: Actually it's four: name/__init__.py, name/__init__.pyc, name.py, and then name.pyc. And just so people have terminology to go with all of this, this search is what the finder does to say whether it can or cannot handle the requested module. Huh, I thought we checked for the directory first and only then checked for the __init__ module within it (hence the generation of ImportWarning when we don't find __init__ after finding a correctly named directory). So a normal miss (i.e. no directory) only needs one stat call. (However, I'll grant that I haven't looked at this particular chunk of code in a fairly long time, so I could easily be wrong). Robert raises a good point about the checks for extension modules as well - we should get an accurate count here so Barry's PEP can pitch the proportional reduction in stat calls accurately. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Le Sun, 28 Feb 2010 21:45:56 +0100, Baptiste Carvello a écrit : bytecode-only in a zip is used by py2exe, cx_freeze and the like, for space reasons. Disabling it would probably hurt them. Source code compresses quite well. I'm not sure it would make much of a difference. AFAIR, when you create a py2exe distribution, what takes most of the place is the interpreter itself as well as any big third-party C libraries such as wxWidgets. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Glenn Linderman wrote: if the command line/runpy can do it, the importer could do it. Just a matter of desire and coding. Whether it is worth pursuing further depends on people's perceptions of kookiness vs. functional and performance considerations. Having .py files around that aren't source text could lead to a lot of confusion, given that most platforms these days decide which application to open for a given file based solely on the filename extension. I wouldn't enjoy trying to open a .py file only to have my text editor blow up because it was actually a binary file. So on balance I think it's a bit too kooky for my taste. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sun, Feb 28, 2010 at 09:45:56PM +0100, Baptiste Carvello wrote: However, making a difference between zipimport and the filesystem importer means the application will stop working if I unzip the library zip file, which is surprising. Unzipping the zip file can be handy when debugging a bug caused by a forgotten module. That difference exists already, the zipimporter will happily run .pyo files inside the zipfile even when you're not running with -O or PYTHONOPTIMIZE. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Floris Bruynooghe wrote: Can't it look for a .py file in the source directory first (1st stat)? When it's there check for the .pyc in the cache directory (2nd stat, magic number encoded in filename), if it's not check for .pyc in the source directory (2nd stat + read for magic number check). Yes, although that would then incur higher stat overheads for people distributing .pyc files. There doesn't seem to be a way of pleasing everyone. This is all assuming that the extra stat calls are actually a problem. Does anyone have any evidence that they would really take significant time compared to loading the module? Once you've looked for one file in a given directory, looking for another one in the same directory ought to be quite fast, since all the relevant directory blocks will be in the filesystem cache. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Mon, 2010-03-01 at 12:35 +1300, Greg Ewing wrote: Yes, although that would then incur higher stat overheads for people distributing .pyc files. There doesn't seem to be a way of pleasing everyone. This is all assuming that the extra stat calls are actually a problem. Does anyone have any evidence that they would really take significant time compared to loading the module? Once you've looked for one file in a given directory, looking for another one in the same directory ought to be quite fast, since all the relevant directory blocks will be in the filesystem cache. We've done a bunch of testing in bzrlib. Basic things are: - statting /is/ expensive *if* you don't use the result. - loading code is the main cost *once* you have a hot disk cache Specifically, stats for files that are *not present* incur page-in costs for the dentries needed to determine the file is absent. In the special case of probing for $name.$ext1, ...$ext2, ...$ext3, you generally hit the same pages and don't incur additional page in costs. (you'll hit the same page in most file systems when you look for the second and third entries). In most file systems stats for files that *are present* also incur a page-in for the inode of the file. If you then do not read the file, this is I/O that doesn't really gain anything. Being able to disable .py file usage completely - so that only foo.pyc and foo/__init__.pyc are probed for, could have a very noticable change in the cold cache startup time. # Startup time for bzr (cold cache): $ drop-caches $ time bzr --no-plugins revno 5061 real0m8.875s user0m0.210s sys 0m0.140s # Hot cache $ time bzr --no-plugins revno 5061 real0m0.307s user0m0.250s sys 0m0.040s (revno is a small command that reads a small amount of data - just enough to trigger demand loading of the core repository layers and so on). strace timings for those two operations: cold cache: $ strace -c bzr --no-plugins revno 5061 % time seconds usecs/call callserrors syscall -- --- --- - - 56.340.04 76 527 read 28.980.020573 9 2273 1905 open 14.430.010248 14 734 625 stat 0.150.000107 0 533 fstat ... hot cache: % time seconds usecs/call callserrors syscall -- --- --- - - 45.100.000368 92 4 getdents 19.490.000159 0 527 read 16.910.000138 1 163 munmap 10.050.82 254 mprotect 8.460.69 0 2273 1905 open 0.000.00 0 8 write 0.000.00 0 367 close 0.000.00 0 734 625 stat ... Cheers, Rob signature.asc Description: This is a digitally signed message part ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Robert Collins wrote: In the special case of probing for $name.$ext1, ...$ext2, ...$ext3, you generally hit the same pages and don't incur additional page in costs. So then looking for a .pyc alongside a .py or vice versa should be almost free, and we shouldn't be worrying about it. hot cache: % time seconds usecs/call callserrors syscall -- --- --- - - 45.100.000368 92 4 getdents 0.000.00 0 734 625 stat Further supporting the idea that stat calls are negligible once the cache is warmed up. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sun, Feb 28, 2010 at 16:31, Greg Ewing greg.ew...@canterbury.ac.nzwrote: Robert Collins wrote: In the special case of probing for $name.$ext1, ...$ext2, ...$ext3, you generally hit the same pages and don't incur additional page in costs. So then looking for a .pyc alongside a .py or vice versa should be almost free, and we shouldn't be worrying about it. But that is making the assumption that all filesystems operate this way (.e.g does NFS have the same performance characteristics?). hot cache: % time seconds usecs/call callserrors syscall -- --- --- - - 45.100.000368 92 4 getdents 0.000.00 0 734 625 stat Further supporting the idea that stat calls are negligible once the cache is warmed up. But that's the point: once it's warmed up. This is not the case when executing a script once every once in a while compared to something bzr where you are most likely going to execute the command multiple times within a small timeframe. -Brett -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sun, Feb 28, 2010 at 12:46, Nick Coghlan ncogh...@gmail.com wrote:
Brett Cannon wrote:
Actually it's four: name/__init__.py, name/__init__.pyc, name.py, and
then name.pyc. And just so people have terminology to go with all of
this, this search is what the finder does to say whether it can or
cannot handle the requested module.
Huh, I thought we checked for the directory first and only then checked
for the __init__ module within it (hence the generation of ImportWarning
when we don't find __init__ after finding a correctly named directory).
So a normal miss (i.e. no directory) only needs one stat call.
(However, I'll grant that I haven't looked at this particular chunk of
code in a fairly long time, so I could easily be wrong).
Robert raises a good point about the checks for extension modules as
well - we should get an accurate count here so Barry's PEP can pitch the
proportional reduction in stat calls accurately.
Here are the details (from Python/import.c:find_module) assuming that
everything has failed to the point of trying for the implicit sys.path
importers:
stat_info = stat(name)
if stat_info.exists and stat_info.is_dir:
if stat(name/__init__.py) || stat(name/__init__.pyc):
load(name)
else:
for ext in ('.so', 'module.so', '.py', 'pyc'): # Windows has an extra
check for .pyw files.
if open(name + ext):
load(name)
So there are a total of five to six depending on the OS (actually, VMS goes
up to eight!) before a search path is considered not to contain a module.
And thanks to doing this I realized importlib is not stat'ing the directory
first which should fail faster than checking for the __init__ files every
time.
-Brett
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
---
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sun, Feb 28, 2010 at 12:45, Baptiste Carvello baptiste...@free.frwrote: Nick Coghlan a écrit : Another option is to remove bytecode-only support from the default filesystem importer, but keep it for zipimport (since the stat call savings don't apply in the latter case). bytecode-only in a zip is used by py2exe, cx_freeze and the like, for space reasons. Disabling it would probably hurt them. However, making a difference between zipimport and the filesystem importer means the application will stop working if I unzip the library zip file, which is surprising. Unzipping the zip file can be handy when debugging a bug caused by a forgotten module. Is it really that hard to unzip a bunch of .pyc files, modify what you need to, and then zip it back up? And if you are given a zip file of only .pyc files you can't really debug anything anyway. -Brett Cheers, Baptiste ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On approximately 2/28/2010 3:22 PM, came the following characters from the keyboard of Greg Ewing: Glenn Linderman wrote: if the command line/runpy can do it, the importer could do it. Just a matter of desire and coding. Whether it is worth pursuing further depends on people's perceptions of kookiness vs. functional and performance considerations. Having .py files around that aren't source text could lead to a lot of confusion, given that most platforms these days decide which application to open for a given file based solely on the filename extension. I wouldn't enjoy trying to open a .py file only to have my text editor blow up because it was actually a binary file. So on balance I think it's a bit too kooky for my taste. I understand your thoughts, but have some rebuttal comments. Mind you, if there is a better solution that can improve performance for both the source+binary and the binary-only distributions, I'm all for it. But in general, I'm all for performance improvements, even if there is some kookiness :) Thankful for Brett's posting of the actual search code fragment. If your text editor blows up because it is binary, it is a sad text editor. If you have .py mapped to a text editor, that's sort of kooky too; I have it mapped to Python. The .py files that are binary would generally be part of an application distribution in binary form, and therefore would be installed in some place like /bin or C:\Program Files ... not the place you'd look for source code, to confuse your text editor. -- Glenn -- http://nevcal.com/ === A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Le Sun, 28 Feb 2010 19:32:09 -0800, Glenn Linderman v+pyt...@g.nevcal.com a écrit : If your text editor blows up because it is binary, it is a sad text editor. If you have .py mapped to a text editor, that's sort of kooky too; I have it mapped to Python. File extensions exist for a reason, even if you find that kooky and have strong ideas about the psychology of text editors. Having some binary files named foobar.py would certainly annoy a lot of people, including me. Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Glenn Linderman wrote: If your text editor blows up because it is binary, it is a sad text editor. Blow up is probably an exaggeration, but even just getting a screen full of gibberish when I think I'm opening a text file is a jarring experience. If you have .py mapped to a text editor, that's sort of kooky too; I have it mapped to Python. On Windows the action for double-clicking is usually mapped to running the file, but there's typically another action such as Open with IDLE or whatever available, and a bytecode file named with .py would allow you to apply that action to it. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Feb 27, 2010, at 9:38 AM, Nick Coghlan wrote: I do like the idea of pulling .pyc only imports out into a separate importer, but would go so far as to suggest keeping them as a command line option rather than as a separately distributed module. One advantage of doing this as a separately distributed module is that it can have its own ecosystem and momentum. Most projects that want this sort of bundling or packaging really want to be shipped with something like py2exe, and I think the folks who want such facilities would be better served by a nice project website for python sealer or python bundler rather than obscure directions for triggering the behavior via options or configuration. Making bytecode loading a feature of interpreter startup, whether it's a config file, a command-line option or an environment variable, is not a great idea. For folks that want to ship a self-contained application, any of these would require an additional customization step, where they need to somehow tell their bundled interpreter to load bytecode. For people trying to ship a self-contained and tamper-unfriendly (since even tamper-resistant would be overstating things) library to relatively non-technical programmers, it opens the door to a whole universe of confusion and FAQs about why the code didn't load. However bytecode-only code loading is facilitated, it should be possible to bootstrap from a vanilla python interpreter running normally, as you may not know you need to load a bytecode-only package at startup. In the stand-alone case there are already plenty of options, and in the library case, shipping a zip file should be fine, since the __init__.py of your package should be plain-text and also able to trigger the activation of the bytecode-only importer. There are already so many ways to ship bytecode already, it doesn't seem too important to support in this one particular configuration (files in a directory, compiled by just importing them, in the same place as .py files). The real problem is providing a seamless transition path for *build* processes, not the Python code itself. Do any of the folks who are currently using this feature have a good idea as to how your build and distribute scripts might easily be updated, perhaps by a 2to3 fixer?___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Greg Ewing greg.ew...@canterbury.ac.nz wrote: Having .py files around that aren't source text could lead to a lot of confusion, given that most platforms these days decide which application to open for a given file based solely on the filename extension. I wouldn't enjoy trying to open a .py file only to have my text editor blow up because it was actually a binary file. So on balance I think it's a bit too kooky for my taste. +1 Add to that the inverse... I will cleanup directories based on the suffix keeping the .py and deleting .pyc and .pyo. Overloading a source file suffix is not good. Larry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Brett Cannon a écrit : However, making a difference between zipimport and the filesystem importer means the application will stop working if I unzip the library zip file, which is surprising. Unzipping the zip file can be handy when debugging a bug caused by a forgotten module. Is it really that hard to unzip a bunch of .pyc files, modify what you need to, and then zip it back up? And if you are given a zip file of only .pyc files you can't really debug anything anyway. Well, this is a micro-use-case, I admit, I only mention it because it's something I've really done. It's only useful for debugging the building process, not the application (so I do have the source at hand), and the only reason for not rezipping is to test more quickly. I can definitely live without it! Cheers, Baptiste ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Brett Cannon wrote: On Fri, Feb 26, 2010 at 20:08, Glenn Linderman v+pyt...@g.nevcal.com I'm not sure why what you did is different than what I did, -M uses runpy which is not directly equivalent to importing. It's actually execution which is different from importing. Direct execution doesn't care about filenames (it inspects the file itself to figure out what it is), while importing cares a great deal. Note that Glenn ran foo.py directly, while Brett did import temp. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Glenn Linderman wrote: But if the technique can work from the command line, it seems the same technique could be re-used in the importer. Not really - we only get away with the fun and games at execution time because __main__ is a bit special (and always has been). Those tricks would be a lot harder to pull off for a normal module import (if they were possible at all - I'm not sure they would be). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Steven D'Aprano wrote: On Sat, 27 Feb 2010 09:09:26 am Brett Cannon wrote: I think it's almost a dis-service to support bytecode-only files as it leads people who are misinformed or simply don't take the time to understand what is contained in a .pyc file into a false sense of security about their code not being easy to examine by someone else. You say that as if it were a bad thing. *wink* Personally, I can't imagine ever wanting to ship a .pyc module without the .py, but since Python already gives people the opportunity to shoot themselves in the foot, meh, we're all adults here. I do recall a poster on comp.lang.python pulling his hair out over a customer who was too big to fire, but who had the obnoxious habit of making random so-called fixes to the poster's .py files, so perhaps byte-code only distribution isn't all bad. I think the use case of keep the user from fiddling casually with our application is a valid one. There's a fairly vast difference between open source file, edit code, hit save and decompile pyc, open decompiled source file, edit code, save next to pyc with correct name. The former makes it easy for folks that know just enough to be dangerous to get themselves in trouble. The latter raises the bar far enough that people with the ability to do it should also know better than to try (or at least, not to call the support line when it doesn't work). I do like the idea of pulling .pyc only imports out into a separate importer, but would go so far as to suggest keeping them as a command line option rather than as a separately distributed module. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Feb 28, 2010, at 01:38 AM, Nick Coghlan wrote: I think the use case of keep the user from fiddling casually with our application is a valid one. Doesn't the existing support for zipimport satisfy that use case already, and probably better so? Heck you can even name your zip file application.dat to really throw naive users off the scent. ;) -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Feb 26, 2010, at 10:59 PM, Michael Foord wrote: There are several companies who currently ship bytecode only. (There was someone on the IronPython mailing list only last week asking if IronPython could support pyc files for this reason). For many pointy-haired-bosses 'some' protection is enough and having Python not support this (out of the box) would be a black mark against Python for them. Would it not be better to ship a zip file with an obfuscated name? Doesn't that satisfy the use case nicely? -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Feb 26, 2010, at 08:30 PM, Ron Adam wrote: It does not make sense (to me) to have byte code only modules and packages in python's lib directory. The whole purpose (as far as I know) is for modules and packages located there to be shared. And as such, the source file becomes a source of documentation. Not supporting bytecode only python modules and packages in pythons lib directory may be good. Actually, it's not the standard library that's the issue, it's third party modules that OS vendors distribute. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Feb 26, 2010, at 05:11 PM, Guido van Rossum wrote: Barry's PEP would fix this even if we kept supporting .pyc-only files: the lingering .pyc files will be in the __pycache__ directory which is *not* searched -- only .pyc files directly in the source directory will be found -- where the PEP will never place them, at least not by default. Exactly so. -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sat, Feb 27, 2010 at 10:56:13AM -0500, Barry Warsaw wrote: On Feb 26, 2010, at 10:59 PM, Michael Foord wrote: There are several companies who currently ship bytecode only. (There was someone on the IronPython mailing list only last week asking if IronPython could support pyc files for this reason). For many pointy-haired-bosses 'some' protection is enough and having Python not support this (out of the box) would be a black mark against Python for them. Would it not be better to ship a zip file with an obfuscated name? Doesn't that satisfy the use case nicely? Sure, we combine that with putting .pyo files inside the zipfile tough (for assert statements and if __debug__ blocks). I'm rather confused about everything proposed by now but would that keep working? Also somewhere else in the thread it seemed like both you and Guido suggested that simply creating a directory with some .pyc (or .pyo I guess) files in would keep working, just by default they won't be written there by python. Or is it that functionality some want to cut because of the doubling of the stat calls? (But even then I'm not convinced that would double the stat calls for normal users, only for those who only ship .pyc files) Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Steven D'Aprano st...@pearwood.info writes: Personally, I can't imagine ever wanting to ship a .pyc module without the .py, but since Python already gives people the opportunity to shoot themselves in the foot, meh, we're all adults here. Not sure I've seen it mentioned in this thread, but for myself, I've certainly used (indirectly) such a distribution many times when packaging applications with py2exe for installation on Windows clients. That puts all the pyc files into a single support zip file from which the application runs. That seems a perfectly useful use case, and not due to any issues with security/obfuscation. The matching interpreter is being packaged with the application, so there's no version worries with the pyc. The files are internal to a zip, so why complicate things with recompiling and writing locally on the user's machine, particularly when on newer versions of Windows the installation directory might not be writable anyway. As long as executing from pyc files continues to work, presumably py2exe can be updated to collect those files from any new cache location during the build process. But I do think it's useful to continue to support executing them directly outside of any new cache location, which it sounds like is the direction being taken. -- David ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Glenn Linderman wrote: What I did was: python -m test ren test.pyc foo.py foo.py and it worked. Source files mentioned on the command line aren't required to have a .py extension. I think what's happening is that the interpreter ignores the filename altogether in that case and examines the contents of the file to figure out what it is, in order to support running .pyc files from the command line. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Le Fri, 26 Feb 2010 14:29:03 -0800, Guido van Rossum a écrit : Byte-code only wasn't always supported. We added it knowing full well it had all those problems (plus, it locks in the Python version), simply because a certain class of developers won't stop asking for it. Their users are apparently too dumb to decode bytecode but smart enough to read source code, even if they don't understand it, and this knowledge could hurt them. The idea that too much knowledge hurts users doesn't sound very Pythonic to me. As I understand it, the people interested in bytecode-only distributions are commercial companies willing to ease support. Why don't they whip up a specialized importer, and perhaps make it available as a recipe or a PyPI module somewhere? The idea that we should provide built-in support for a stupid (non-)security mechanism sounds insane to me. Finally, the sight of commercial companies not being able to do their work and begging open source projects to do it for them makes me *yawn*. If you aren't proficient or motivated enough to build your own internal commodities, perhaps you shouldn't do claim to do business at all. regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On 28/02/2010 01:22, Antoine Pitrou wrote: Le Fri, 26 Feb 2010 14:29:03 -0800, Guido van Rossum a écrit : Byte-code only wasn't always supported. We added it knowing full well it had all those problems (plus, it locks in the Python version), simply because a certain class of developers won't stop asking for it. Their users are apparently too dumb to decode bytecode but smart enough to read source code, even if they don't understand it, and this knowledge could hurt them. The idea that too much knowledge hurts users doesn't sound very Pythonic to me. As I understand it, the people interested in bytecode-only distributions are commercial companies willing to ease support. Why don't they whip up a specialized importer, and perhaps make it available as a recipe or a PyPI module somewhere? The idea that we should provide built-in support for a stupid (non-)security mechanism sounds insane to me. Finally, the sight of commercial companies not being able to do their work and begging open source projects to do it for them makes me *yawn*. If you aren't proficient or motivated enough to build your own internal commodities, perhaps you shouldn't do claim to do business at all. Well if we'd *never* had this feature this argument would be very strong indeed. On the other hand if we want them to switch to Python 3 - but by the way we cut one of the features you rely on, but don't worry all you have to do is recode it yourself - doesn't make a very convincing argument. Michael Foord regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Le Sun, 28 Feb 2010 01:25:38 +, Michael Foord fuzzy...@voidspace.org.uk a écrit : Well if we'd *never* had this feature this argument would be very strong indeed. On the other hand if we want them to switch to Python 3 - but by the way we cut one of the features you rely on, but don't worry all you have to do is recode it yourself - doesn't make a very convincing argument. I understand it. On the other hand, it is certainly one of the least important issues involved in porting to py3k. (even for those people who liked the feature) And I think the prospect of a slight simplification (or de-complexification) of the import machinery is an important selling point. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Floris Bruynooghe wrote: (But even then I'm not convinced that would double the stat calls for normal users, only for those who only ship .pyc files) It would increase the number of stat calls for normal users by 50%. You would need to look for a .pyc in the source directory, then .py in the source directory and .pyc in the cache directory. That's compared to two stat calls currently, for .py and .pyc. A solution might be to look for the presence of the cache directory, and only look for a .pyc in the source directory if there is no cache directory. Testing for the cache directory would only have to be done once per package and the result remembered, so it would add very little overhead. -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Glenn Linderman wrote: Thanks for the explanation. Brett mentioned something like runpy vs import using different techniques. Which is OK, I guess, but if the command line/runpy can do it, the importer could do it. Just a matter of desire and coding. Whether it is worth pursuing further depends on people's perceptions of kookiness vs. functional and performance considerations. As I said previously, don't underestimate how different __main__ is from everything else. The most obvious difference is that the code for __main__ is executed without holding the import lock, but there are other important differences as well (such as the module object being created directly by the interpreter startup sequence and hence a lot of the import machinery being bypassed). Even the -m switch doesn't really follow the normal import paths (it just finds the code object for the named module and then runs it directly instead of importing it). Direct execution starts with a filename (or a module name when using -m) then works out how to execute it as __main__. Importing starts with a module name, tries to find a matching filename and create the corresponding module. The different starting points and the different end goals affect the assumptions that are made while the interpreter figures out what it needs to do. The behaviour of runpy is different from import precisely because it aims to mimic execution of __main__ rather than a normal import. If there weren't quite so many semantic differences between direct execution and normal import, the module would have been a lot easier to write :) Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Thu, Feb 25, 2010 at 16:13, Greg Ewing greg.ew...@canterbury.ac.nzwrote: Michael Foord wrote: I thought we agreed at the language summit that if a .pyc was in the place of the source file it *could* be imported from - making pyc only distributions possible. Ah, that's okay, then. Sorry about the panic! Michael is right about what as discussed at the language summit, but Barry means what he says; if you look at the PEP as it currently stands it does not support bytecode-only modules. Barry and I discussed how to implement the PEP at PyCon after the summit and supporting bytecode-only modules quickly began to muck with the semantics and made it harder to explain (i.e. what to set __file__ vs. __compiled__ based on what is or is not available and how to properly define get_paths for loaders). But a benefit of no longer supporting bytecode-only modules by default is it cuts back on possible stat calls which slows down Python's startup time (a complaint I hear a lot). Performance issues become even more acute if you try to come up with even a remotely proper way to have backwards-compatible support in importlib for its ABCs w/o forcing caching on all implementors of the ABCs. As for having a dependency on a loader, I don't see how that is obscure; it's just a dependency your package has that you handle at install-time. And personally, I don't see what bytecode-only modules buy you. The obfuscation argument is bunk as we all know. Bytecode contains so much data that disassembling it gives you a very clear picture of what the original code was like. I think it's almost a dis-service to support bytecode-only files as it leads people who are misinformed or simply don't take the time to understand what is contained in a .pyc file into a false sense of security about their code not being easy to examine by someone else. The only perk I can see is space-saving, but that's dangerous as that ties you to a specific VM with a specific magic number (let alone that it leads to people tying themselves to CPython and ignoring the other VMs that simply do not support bytecode). -Brett -- Greg ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Fri, Feb 26, 2010 at 2:09 PM, Brett Cannon br...@python.org wrote: And personally, I don't see what bytecode-only modules buy you. The obfuscation argument is bunk as we all know. Bytecode contains so much data that disassembling it gives you a very clear picture of what the original code was like. I think it's almost a dis-service to support bytecode-only files as it leads people who are misinformed or simply don't take the time to understand what is contained in a .pyc file into a false sense of security about their code not being easy to examine by someone else. Byte-code only wasn't always supported. We added it knowing full well it had all those problems (plus, it locks in the Python version), simply because a certain class of developers won't stop asking for it. Their users are apparently too dumb to decode bytecode but smart enough to read source code, even if they don't understand it, and this knowledge could hurt them. Presumably users smart enough to decode bytecode will know enough not to hurt themselves. Maybe Greg's and my response to the mention of dropping this feature is too strong -- after all we're both dinosaurs. And maybe the developers who want the feature can write their own loader. But given that this feature takes an entirely different path through import.c anyway, I still don't see how dropping it is necessary in order to implement the PEP. If you have separate motivation to drop the feature, you should deprecate it properly. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Feb 26, 2010, at 02:09 PM, Brett Cannon wrote: But a benefit of no longer supporting bytecode-only modules by default is it cuts back on possible stat calls which slows down Python's startup time (a complaint I hear a lot). Performance issues become even more acute if you try to come up with even a remotely proper way to have backwards-compatible support in importlib for its ABCs w/o forcing caching on all implementors of the ABCs. And personally, I don't see what bytecode-only modules buy you. The obfuscation argument is bunk as we all know. Brett really hits the nail on the head, and yes I'm sorry for not being clear about what we discussed this at Pycon meant. The we being Brett and I of course (and Chris Withers IIRC). Bytecode-only deployments are a bit of a sham, and definitely a minority use case, so why should all of Python pay for the extra stat calls to support this by default? How many people would actually be hurt if this wasn't available out of the box, especially since you can still support it if you really want it and can't convince your manager that it provides essentially zero useful obfuscation of your code? I say this having been down that road myself with a previous employer. Management was pretty adamant about wanting this until I explained how easy it was to defeat and convinced them that the engineering resources to do it were better spent elsewhere. Having said that, I'd be all for including a reference implementation of a bytecode-only loader in the PEP for demonstration purposes. Greg, would you like to contribute that? -Barry signature.asc Description: PGP signature ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Fri, Feb 26, 2010 at 14:29, Guido van Rossum gu...@python.org wrote: On Fri, Feb 26, 2010 at 2:09 PM, Brett Cannon br...@python.org wrote: And personally, I don't see what bytecode-only modules buy you. The obfuscation argument is bunk as we all know. Bytecode contains so much data that disassembling it gives you a very clear picture of what the original code was like. I think it's almost a dis-service to support bytecode-only files as it leads people who are misinformed or simply don't take the time to understand what is contained in a .pyc file into a false sense of security about their code not being easy to examine by someone else. Byte-code only wasn't always supported. We added it knowing full well it had all those problems (plus, it locks in the Python version), simply because a certain class of developers won't stop asking for it. Their users are apparently too dumb to decode bytecode but smart enough to read source code, even if they don't understand it, and this knowledge could hurt them. Presumably users smart enough to decode bytecode will know enough not to hurt themselves. Maybe it should be made optional much like the talk of frozen modules eventually becoming an optional thing. Maybe Greg's and my response to the mention of dropping this feature is too strong -- after all we're both dinosaurs. And maybe the developers who want the feature can write their own loader. We could also provide if necessary. But given that this feature takes an entirely different path through import.c anyway, I still don't see how dropping it is necessary in order to implement the PEP. It's not necessary at all. I think what Barry was going for was simply cleaning up semantics once instead of having to drag it out. If you have separate motivation to drop the feature, you should deprecate it properly. Fine by me. It would be easy enough to raise ImportWarning in the bytecode-only case if Barry decides to push for this. Here is a question for Barry to think about if he decides to move forward with all of this: would mixed support for both bytecode-only and source/bytecode be required for the same directory, or could it be one or the other but not both? Differing semantics based on what is found in the directory would make the path hook more expensive (which is a one-time cost per directory), but it would cut stat calls in the finder in half (which is a cost made per import). -Brett -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On 26/02/2010 22:09, Brett Cannon wrote: On Thu, Feb 25, 2010 at 16:13, Greg Ewing greg.ew...@canterbury.ac.nz mailto:greg.ew...@canterbury.ac.nz wrote: Michael Foord wrote: I thought we agreed at the language summit that if a .pyc was in the place of the source file it *could* be imported from - making pyc only distributions possible. Ah, that's okay, then. Sorry about the panic! Michael is right about what as discussed at the language summit, but Barry means what he says; if you look at the PEP as it currently stands it does not support bytecode-only modules. Barry and I discussed how to implement the PEP at PyCon after the summit and supporting bytecode-only modules quickly began to muck with the semantics and made it harder to explain (i.e. what to set __file__ vs. __compiled__ based on what is or is not available and how to properly define get_paths for loaders). But a benefit of no longer supporting bytecode-only modules by default is it cuts back on possible stat calls which slows down Python's startup time (a complaint I hear a lot). Performance issues become even more acute if you try to come up with even a remotely proper way to have backwards-compatible support in importlib for its ABCs w/o forcing caching on all implementors of the ABCs. As for having a dependency on a loader, I don't see how that is obscure; it's just a dependency your package has that you handle at install-time. And personally, I don't see what bytecode-only modules buy you. The obfuscation argument is bunk as we all know. Bytecode contains so much data that disassembling it gives you a very clear picture of what the original code was like. Well, understanding bytecode is *still* requires a higher level of understanding than the *majority* of Python programmers. Added to which there are no widely available tools that *I'm* aware of for decompiling recent versions of Python (decompyle worked up to Python 2.4 but then went closed source as a commercial service [1]. The situation is analagous to .NET assemblies by the way (which *can* be trivially decompiled by several widely available tools). Having a non-source distribution prevents your users from changing things and then calling you for support without them having to go to a lot more effort than it is worth. There are several companies who currently ship bytecode only. (There was someone on the IronPython mailing list only last week asking if IronPython could support pyc files for this reason). For many pointy-haired-bosses 'some' protection is enough and having Python not support this (out of the box) would be a black mark against Python for them. I think it's almost a dis-service to support bytecode-only files as it leads people who are misinformed or simply don't take the time to understand what is contained in a .pyc file into a false sense of security about their code not being easy to examine by someone else. For many use-cases some protection is enough. After all *any* DRM or source-code obfuscation is breakable in the medium / long term - so just enough to discourage the casual looker is probably sufficient. The fact that bytecode only distributions exist speaks to that. Whether you believe that allowing companies who ship bytecode is a disservice to them or not is fundamentally irrelevant. If they believe it is a service to them then it is... :-) As you can tell, I would be disappointed to see bytecode only distributions be removed from the out-of-the-box functionality. All the best, Michael The only perk I can see is space-saving, but that's dangerous as that ties you to a specific VM with a specific magic number (let alone that it leads to people tying themselves to CPython and ignoring the other VMs that simply do not support bytecode). [1] http://www.crazy-compilers.com/decompyle/ -Brett -- Greg ___ Python-Dev mailing list Python-Dev@python.org mailto:Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. ___ Python-Dev mailing
Re: [Python-Dev] __file__
On approximately 2/26/2010 2:55 PM, came the following characters from the keyboard of Brett Cannon: Maybe Greg's and my response to the mention of dropping this feature is too strong -- after all we're both dinosaurs. And maybe the developers who want the feature can write their own loader. We could also provide if necessary. So if the implementation stores .pyc by default in a version-specific place, then it seems there are only two things needed to make a python byte-code only distribution... 1) rename all the .pyc to .py 2) packaging When a .pyc is renamed to .py, Python (3.1 at least) recognizes and uses it... I assume by design, rather than accident, but I don't know the history. I didn't experiment to discover what __file__ and __cached__ get set to in this case (especially since I don't have a version with the latter :) ). I speculate that packaging a distribution in this manner would be slightly different that how it is currently done, but I also suspect that it would avoid the same half of the stat calls, to aid performance. -- Glenn -- http://nevcal.com/ === A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
The one issue I thought would be resolved by not easily allowing .pyc-only distributions is the case when you rename a file (say module.py to newmodule.py) and there is a module.pyc laying around, and you don't get the ImportError you would expect from import module -- and to make it worse everything basically works, except there's two versions of the module that slowly become different. This regularly causes problems for me, and those problems would get more common and obscure if the pyc files were stashed away in a more invisible location. I can't even tell what the current proposal is; maybe this is resolved? If distributing bytecode required renaming pyc files to .py as Glenn suggested that would resolve the problem quite nicely from my perspective. (Frankly I find the whole use case for distributing bytecodes a bit specious, but whatever.) -- Ian Bicking | http://blog.ianbicking.org | http://twitter.com/ianbicking ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Fri, Feb 26, 2010 at 16:58, Ian Bicking i...@colorstudy.com wrote: The one issue I thought would be resolved by not easily allowing .pyc-only distributions is the case when you rename a file (say module.py to newmodule.py) and there is a module.pyc laying around, and you don't get the ImportError you would expect from import module -- and to make it worse everything basically works, except there's two versions of the module that slowly become different. Yes, that problem would go away if bytecode-only modules were no longer supported. This regularly causes problems for me, and those problems would get more common and obscure if the pyc files were stashed away in a more invisible location. That has never been an issue with this proposal. The bytecode pulled from the __pycache__ directory only occurs if source exists. What we have been discussing is whether bytecode-only files in the directory of a package or something exists. -Brett I can't even tell what the current proposal is; maybe this is resolved? If distributing bytecode required renaming pyc files to .py as Glenn suggested that would resolve the problem quite nicely from my perspective. (Frankly I find the whole use case for distributing bytecodes a bit specious, but whatever.) -- Ian Bicking | http://blog.ianbicking.org | http://twitter.com/ianbicking ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Fri, Feb 26, 2010 at 4:58 PM, Ian Bicking i...@colorstudy.com wrote: The one issue I thought would be resolved by not easily allowing .pyc-only distributions is the case when you rename a file (say module.py to newmodule.py) and there is a module.pyc laying around, and you don't get the ImportError you would expect from import module -- and to make it worse everything basically works, except there's two versions of the module that slowly become different. This regularly causes problems for me, and those problems would get more common and obscure if the pyc files were stashed away in a more invisible location. I can't even tell what the current proposal is; maybe this is resolved? If distributing bytecode required renaming pyc files to .py as Glenn suggested that would resolve the problem quite nicely from my perspective. (Frankly I find the whole use case for distributing bytecodes a bit specious, but whatever.) Barry's PEP would fix this even if we kept supporting .pyc-only files: the lingering .pyc files will be in the __pycache__ directory which is *not* searched -- only .pyc files directly in the source directory will be found -- where the PEP will never place them, at least not by default. -- --Guido van Rossum (python.org/~guido) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Fri, Feb 26, 2010 at 15:35, Glenn Linderman v+pyt...@g.nevcal.comv%2bpyt...@g.nevcal.com wrote: On approximately 2/26/2010 2:55 PM, came the following characters from the keyboard of Brett Cannon: Maybe Greg's and my response to the mention of dropping this feature is too strong -- after all we're both dinosaurs. And maybe the developers who want the feature can write their own loader. We could also provide if necessary. So if the implementation stores .pyc by default in a version-specific place, then it seems there are only two things needed to make a python byte-code only distribution... 1) rename all the .pyc to .py 2) packaging When a .pyc is renamed to .py, Python (3.1 at least) recognizes and uses it... I assume by design, rather than accident, but I don't know the history. This does not work for me (nor should it): touch temp.py python3 -c import temp rm temp.py mv temp.pyc temp.py python3 -c import temp Traceback (most recent call last): File string, line 1, in module File temp.py, line 2 SyntaxError: Non-UTF-8 code starting with '\x95' in file temp.py on line 2, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details -Brett I didn't experiment to discover what __file__ and __cached__ get set to in this case (especially since I don't have a version with the latter :) ). I speculate that packaging a distribution in this manner would be slightly different that how it is currently done, but I also suspect that it would avoid the same half of the stat calls, to aid performance. -- Glenn -- http://nevcal.com/ === A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Feb 26, 2010, at 5:59 PM, Michael Foord wrote: On 26/02/2010 22:09, Brett Cannon wrote: On Thu, Feb 25, 2010 at 16:13, Greg Ewing greg.ew...@canterbury.ac.nz wrote: Michael Foord wrote: I thought we agreed at the language summit that if a .pyc was in the place of the source file it *could* be imported from - making pyc only distributions possible. Ah, that's okay, then. Sorry about the panic! Michael is right about what as discussed at the language summit, but Barry means what he says; if you look at the PEP as it currently stands it does not support bytecode-only modules. Barry and I discussed how to implement the PEP at PyCon after the summit and supporting bytecode-only modules quickly began to muck with the semantics and made it harder to explain (i.e. what to set __file__ vs. __compiled__ based on what is or is not available and how to properly define get_paths for loaders). But a benefit of no longer supporting bytecode-only modules by default is it cuts back on possible stat calls which slows down Python's startup time (a complaint I hear a lot). Performance issues become even more acute if you try to come up with even a remotely proper way to have backwards-compatible support in importlib for its ABCs w/o forcing caching on all implementors of the ABCs. As for having a dependency on a loader, I don't see how that is obscure; it's just a dependency your package has that you handle at install-time. And personally, I don't see what bytecode-only modules buy you. The obfuscation argument is bunk as we all know. Bytecode contains so much data that disassembling it gives you a very clear picture of what the original code was like. Well, understanding bytecode is *still* requires a higher level of understanding than the *majority* of Python programmers. Added to which there are no widely available tools that *I'm* aware of for decompiling recent versions of Python (decompyle worked up to Python 2.4 but then went closed source as a commercial service [1]. The situation is analagous to .NET assemblies by the way (which *can* be trivially decompiled by several widely available tools). Having a non-source distribution prevents your users from changing things and then calling you for support without them having to go to a lot more effort than it is worth. There are several companies who currently ship bytecode only. (There was someone on the IronPython mailing list only last week asking if IronPython could support pyc files for this reason). For many pointy- haired-bosses 'some' protection is enough and having Python not support this (out of the box) would be a black mark against Python for them. We ship bytecode only, basically for the reason Michael states here (keeping support costs under control from ambitious users). I think it's almost a dis-service to support bytecode-only files as it leads people who are misinformed or simply don't take the time to understand what is contained in a .pyc file into a false sense of security about their code not being easy to examine by someone else. For many use-cases some protection is enough. After all *any* DRM or source-code obfuscation is breakable in the medium / long term - so just enough to discourage the casual looker is probably sufficient. The fact that bytecode only distributions exist speaks to that. Right. We're more concerned with not having users muck with stuff than with keeping the implementation a secret, although having a bit of obfuscation doesn't hurt. Whether you believe that allowing companies who ship bytecode is a disservice to them or not is fundamentally irrelevant. If they believe it is a service to them then it is... :-) As you can tell, I would be disappointed to see bytecode only distributions be removed from the out-of-the-box functionality. +1 Doug ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Sat, 27 Feb 2010 09:09:26 am Brett Cannon wrote: I think it's almost a dis-service to support bytecode-only files as it leads people who are misinformed or simply don't take the time to understand what is contained in a .pyc file into a false sense of security about their code not being easy to examine by someone else. You say that as if it were a bad thing. *wink* Personally, I can't imagine ever wanting to ship a .pyc module without the .py, but since Python already gives people the opportunity to shoot themselves in the foot, meh, we're all adults here. I do recall a poster on comp.lang.python pulling his hair out over a customer who was too big to fire, but who had the obnoxious habit of making random so-called fixes to the poster's .py files, so perhaps byte-code only distribution isn't all bad. But I don't care much either way. -- Steven D'Aprano ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Fri, Feb 26, 2010 at 17:20, Doug Hellmann doug.hellm...@gmail.comwrote: On Feb 26, 2010, at 5:59 PM, Michael Foord wrote: On 26/02/2010 22:09, Brett Cannon wrote: On Thu, Feb 25, 2010 at 16:13, Greg Ewing greg.ew...@canterbury.ac.nzwrote: Michael Foord wrote: I thought we agreed at the language summit that if a .pyc was in the place of the source file it *could* be imported from - making pyc only distributions possible. Ah, that's okay, then. Sorry about the panic! Michael is right about what as discussed at the language summit, but Barry means what he says; if you look at the PEP as it currently stands it does not support bytecode-only modules. Barry and I discussed how to implement the PEP at PyCon after the summit and supporting bytecode-only modules quickly began to muck with the semantics and made it harder to explain (i.e. what to set __file__ vs. __compiled__ based on what is or is not available and how to properly define get_paths for loaders). But a benefit of no longer supporting bytecode-only modules by default is it cuts back on possible stat calls which slows down Python's startup time (a complaint I hear a lot). Performance issues become even more acute if you try to come up with even a remotely proper way to have backwards-compatible support in importlib for its ABCs w/o forcing caching on all implementors of the ABCs. As for having a dependency on a loader, I don't see how that is obscure; it's just a dependency your package has that you handle at install-time. And personally, I don't see what bytecode-only modules buy you. The obfuscation argument is bunk as we all know. Bytecode contains so much data that disassembling it gives you a very clear picture of what the original code was like. Well, understanding bytecode is *still* requires a higher level of understanding than the *majority* of Python programmers. Added to which there are no widely available tools that *I'm* aware of for decompiling recent versions of Python (decompyle worked up to Python 2.4 but then went closed source as a commercial service [1]. The situation is analagous to .NET assemblies by the way (which *can* be trivially decompiled by several widely available tools). Having a non-source distribution prevents your users from changing things and then calling you for support without them having to go to a lot more effort than it is worth. There are several companies who currently ship bytecode only. (There was someone on the IronPython mailing list only last week asking if IronPython could support pyc files for this reason). For many pointy-haired-bosses 'some' protection is enough and having Python not support this (out of the box) would be a black mark against Python for them. We ship bytecode only, basically for the reason Michael states here (keeping support costs under control from ambitious users). I think it's almost a dis-service to support bytecode-only files as it leads people who are misinformed or simply don't take the time to understand what is contained in a .pyc file into a false sense of security about their code not being easy to examine by someone else. For many use-cases some protection is enough. After all *any* DRM or source-code obfuscation is breakable in the medium / long term - so just enough to discourage the casual looker is probably sufficient. The fact that bytecode only distributions exist speaks to that. Right. We're more concerned with not having users muck with stuff than with keeping the implementation a secret, although having a bit of obfuscation doesn't hurt. Whether you believe that allowing companies who ship bytecode is a disservice to them or not is fundamentally irrelevant. If they believe it is a service to them then it is... :-) As you can tell, I would be disappointed to see bytecode only distributions be removed from the out-of-the-box functionality. +1 So what is the burden of including a single source file that added the support to load from bytecode-only modules? I am not saying you shouldn't be able to have this functionality, just that I personally don't want to pay for the overhead (both performance-wise and development-wise) by default just because you and some other people want this functionality for some clients. -Brett ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
Barry Warsaw wrote: On Feb 26, 2010, at 02:09 PM, Brett Cannon wrote: But a benefit of no longer supporting bytecode-only modules by default is it cuts back on possible stat calls which slows down Python's startup time (a complaint I hear a lot). Performance issues become even more acute if you try to come up with even a remotely proper way to have backwards-compatible support in importlib for its ABCs w/o forcing caching on all implementors of the ABCs. And personally, I don't see what bytecode-only modules buy you. The obfuscation argument is bunk as we all know. Brett really hits the nail on the head, and yes I'm sorry for not being clear about what we discussed this at Pycon meant. The we being Brett and I of course (and Chris Withers IIRC). Bytecode-only deployments are a bit of a sham, and definitely a minority use case, so why should all of Python pay for the extra stat calls to support this by default? How many people would actually be hurt if this wasn't available out of the box, especially since you can still support it if you really want it and can't convince your manager that it provides essentially zero useful obfuscation of your code? I say this having been down that road myself with a previous employer. Management was pretty adamant about wanting this until I explained how easy it was to defeat and convinced them that the engineering resources to do it were better spent elsewhere. Having said that, I'd be all for including a reference implementation of a bytecode-only loader in the PEP for demonstration purposes. Greg, would you like to contribute that? -Barry Micheal Foords view point on this strikes me as the most realistic. Some people do find it to be a value for their particular needs and circumstance. Michael Foord Wrote: For many use-cases some protection is enough. After all *any* DRM or source-code obfuscation is breakable in the medium / long term - so just enough to discourage the casual looker is probably sufficient. The fact that bytecode only distributions exist speaks to that. Whether you believe that allowing companies who ship bytecode is a disservice to them or not is fundamentally irrelevant. If they believe it is a service to them then it is... :-) To possibly qualify it a bit more: It does not make sense (to me) to have byte code only modules and packages in python's lib directory. The whole purpose (as far as I know) is for modules and packages located there to be shared. And as such, the source file becomes a source of documentation. Not supporting bytecode only python modules and packages in pythons lib directory may be good. For python programs located and installed elsewhere I think Michaels view point is applicable. For some files that are not meant to be shared, some form of discouragement can be a feature. Ron Adam ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On approximately 2/26/2010 5:13 PM, came the following characters from the keyboard of Brett Cannon: On Fri, Feb 26, 2010 at 15:35, Glenn Linderman v+pyt...@g.nevcal.com mailto:v%2bpyt...@g.nevcal.com wrote: On approximately 2/26/2010 2:55 PM, came the following characters from the keyboard of Brett Cannon: Maybe Greg's and my response to the mention of dropping this feature is too strong -- after all we're both dinosaurs. And maybe the developers who want the feature can write their own loader. We could also provide if necessary. So if the implementation stores .pyc by default in a version-specific place, then it seems there are only two things needed to make a python byte-code only distribution... 1) rename all the .pyc to .py 2) packaging When a .pyc is renamed to .py, Python (3.1 at least) recognizes and uses it... I assume by design, rather than accident, but I don't know the history. This does not work for me (nor should it): touch temp.py python3 -c import temp rm temp.py mv temp.pyc temp.py python3 -c import temp Traceback (most recent call last): File string, line 1, in module File temp.py, line 2 SyntaxError: Non-UTF-8 code starting with '\x95' in file temp.py on line 2, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details -Brett I'll admit to not doing exhaustive testing, but I'll not admit to not doing any testing... because it was sort of a wild idea. Someone else called it kooky, which is fair. What I did was: python -m test ren test.pyc foo.py foo.py and it worked. Then I posted, knowing that I'd also tested, the other day, several .py into a .zip named .py, and once that worked, then I changed to putting all .pyc into the .zip named .py and that worked too... including imports of the several modules from the __main__.pyc. Of course, all those were still named .pyc inside the .zip named .py. So I'm not sure what the difference is... .pyc as .py works from the command line, but not from import? Some specialty because of using -c ? I'd guess the technique could be made to work, probably not require extensive changes, if Python developers wanted to make it work. I think it could be efficient and that same someone that called it kooky admitted it would solve their use case, at least. I'm not sure why what you did is different than what I did, nor why you state without justification that it shouldn't work... I might be able to figure out the former if I spend enough time with the documentation, if it is documented, but I'm too new to Python to understand the latter without explanation. Could you supply at least the latter explanation? I'd like to understand the issue here, whether or not the kooky idea goes forward. -- Glenn -- http://nevcal.com/ === A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] __file__
On Fri, Feb 26, 2010 at 20:08, Glenn Linderman v+pyt...@g.nevcal.comv%2bpyt...@g.nevcal.com wrote: On approximately 2/26/2010 5:13 PM, came the following characters from the keyboard of Brett Cannon: On Fri, Feb 26, 2010 at 15:35, Glenn Linderman v+pyt...@g.nevcal.comv%2bpyt...@g.nevcal.commailto: v%2bpyt...@g.nevcal.com v%252bpyt...@g.nevcal.com wrote: On approximately 2/26/2010 2:55 PM, came the following characters from the keyboard of Brett Cannon: Maybe Greg's and my response to the mention of dropping this feature is too strong -- after all we're both dinosaurs. And maybe the developers who want the feature can write their own loader. We could also provide if necessary. So if the implementation stores .pyc by default in a version-specific place, then it seems there are only two things needed to make a python byte-code only distribution... 1) rename all the .pyc to .py 2) packaging When a .pyc is renamed to .py, Python (3.1 at least) recognizes and uses it... I assume by design, rather than accident, but I don't know the history. This does not work for me (nor should it): touch temp.py python3 -c import temp rm temp.py mv temp.pyc temp.py python3 -c import temp Traceback (most recent call last): File string, line 1, in module File temp.py, line 2 SyntaxError: Non-UTF-8 code starting with '\x95' in file temp.py on line 2, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details -Brett I'll admit to not doing exhaustive testing, but I'll not admit to not doing any testing... because it was sort of a wild idea. Someone else called it kooky, which is fair. What I did was: python -m test ren test.pyc foo.py foo.py and it worked. Then I posted, knowing that I'd also tested, the other day, several .py into a .zip named .py, and once that worked, then I changed to putting all .pyc into the .zip named .py and that worked too... including imports of the several modules from the __main__.pyc. Of course, all those were still named .pyc inside the .zip named .py. So I'm not sure what the difference is... .pyc as .py works from the command line, but not from import? Some specialty because of using -c ? I'd guess the technique could be made to work, probably not require extensive changes, if Python developers wanted to make it work. I think it could be efficient and that same someone that called it kooky admitted it would solve their use case, at least. I'm not sure why what you did is different than what I did, -M uses runpy which is not directly equivalent to importing. nor why you state without justification that it shouldn't work... It just is not supposed to happen that way. Masquerading a bytecode file as a source file shouldn't work; imp.get_suffixes() controls how files should be interpreted based on their file extension. -Brett I might be able to figure out the former if I spend enough time with the documentation, if it is documented, but I'm too new to Python to understand the latter without explanation. Could you supply at least the latter explanation? I'd like to understand the issue here, whether or not the kooky idea goes forward. -- Glenn -- http://nevcal.com/ === A protocol is complete when there is nothing left to remove. -- Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
