Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 19 February 2015 at 07:16, Paul Moore p.f.mo...@gmail.com wrote: On 18 February 2015 at 20:48, Jim J. Jewett jimjjew...@gmail.com wrote: Barry Warsaw wrote: I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible. Paul Moore wrote: I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately. Are you just looking for http://www.iana.org/assignments/media-types/media-types.xhtml and its references, including the registration procedures http://tools.ietf.org/html/rfc6838#section-4.2.5 and the application form at http://www.iana.org/form/media-types ? That covers mime types, but not file extensions, so it's not really what *I* thought Barry was talking about. FWIW, registering the vnd.python MIME prefix with IANA has been vaguely kicking around on my todo list since we switched the PyPA metadata PEPs over to using JSON. If anyone did decide to follow up on that idea, the PSF Infrastructure working group mailing list would likely be a suitable contact address to use (Infra already handles relationships with other internet registries for DNS et al, and I can't think of another group better suited to the task). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Wed, Feb 18, 2015 at 10:16 PM, Paul Moore p.f.mo...@gmail.com wrote: On 18 February 2015 at 20:48, Jim J. Jewett jimjjew...@gmail.com wrote: Barry Warsaw wrote: I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible. Paul Moore wrote: I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately. Are you just looking for http://www.iana.org/assignments/media-types/media-types.xhtml and its references, including the registration procedures http://tools.ietf.org/html/rfc6838#section-4.2.5 and the application form at http://www.iana.org/form/media-types ? That covers mime types, but not file extensions, so it's not really what *I* thought Barry was talking about. Actually, I guess the first question is whether .py is reserved for Python scripts. If it is, then certainly /pyz should be reserved in a similar way for Python Zip Applications. If there's no formal registration of .py, then it seems a bit unreasonable to bother for .pyz... Also, I don't think reserving anything is something I, as an individual (and specifically a non-Unix user) should do. It probably should be handled by the PSF, as the process seems to need a contact email address... There is no extension registry for Unixy systems, because extensions traditionally don't matter. File types are identified by their contents; there is a magic* database that maps file contents to types (MIME types, nowadays). The `file` command is the CLI interface for that. Some programs will fall back on the extension if that doesn't give a result, but that would depend on the program. For a pyz file to run when you open it (for most values of open), it needs to have the executable bit set, and have a shebang line. * /usr/share/misc/magic – it originally contained just magic numbers, such as those you might find at the beginning of a .pyc file. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Wed, Feb 18, 2015 at 4:16 PM, Paul Moore p.f.mo...@gmail.com wrote: On 18 February 2015 at 20:48, Jim J. Jewett jimjjew...@gmail.com wrote: Barry Warsaw wrote: I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, ... Are you just looking for http://www.iana.org/assignments/media-types/media-types.xhtml and ... That covers mime types, but not file extensions, so it's not really what *I* thought Barry was talking about. Question 13 at http://www.iana.org/form/media-types asks for additional information, and specifically calls out Magic Number and File Extension, among others. I doubt there is any more official repository for file extension meaning within MIME or unix. Also, I don't think reserving anything is something I, as an individual (and specifically a non-Unix user) should do. It probably should be handled by the PSF, as the process seems to need a contact email address... Ideally, it would be a long-lasting organizational address, such as pep-edi...@python.org. But often, it isn't. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 19 February 2015 at 18:11, Jim J. Jewett jimjjew...@gmail.com wrote: Also, I don't think reserving anything is something I, as an individual (and specifically a non-Unix user) should do. It probably should be handled by the PSF, as the process seems to need a contact email address... Ideally, it would be a long-lasting organizational address, such as pep-edi...@python.org. But often, it isn't. Given that .py isn't on that list in any form I could locate, I don't think it's reasonable to add .pyz. If someone wants to propose adding both, I'd be fine with that, but it can be done independently of PEP 441. I'll add a note to the PEP to that effect. On that note, I intend to post the latest version of the PEP, and the reference implementation, tomorrow. I'll get the official version of the PEP on python.org updated at the same time. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
Barry Warsaw wrote: I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible. Paul Moore wrote: I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately. Are you just looking for http://www.iana.org/assignments/media-types/media-types.xhtml and its references, including the registration procedures http://tools.ietf.org/html/rfc6838#section-4.2.5 and the application form at http://www.iana.org/form/media-types ? -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 18, 2015, at 09:16 PM, Paul Moore wrote: That covers mime types, but not file extensions, so it's not really what *I* thought Barry was talking about. I was actually thinking about both. :) The Wikipedia page has several external links to collections of file extensions, but I don't know that there's *an* official registry. http://en.wikipedia.org/wiki/File_extension Picking FileInfo.com at random, and searching for a few .py* extensions, you find the usual suspects, plus a few others, but all are Python related, and there's no .pyz extension currently registered. Also, I don't think reserving anything is something I, as an individual (and specifically a non-Unix user) should do. It probably should be handled by the PSF, as the process seems to need a contact email address... Probably so. Seems like they could pick one or more of these registries and register pyz for this purpose. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 18 February 2015 at 20:48, Jim J. Jewett jimjjew...@gmail.com wrote: Barry Warsaw wrote: I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible. Paul Moore wrote: I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately. Are you just looking for http://www.iana.org/assignments/media-types/media-types.xhtml and its references, including the registration procedures http://tools.ietf.org/html/rfc6838#section-4.2.5 and the application form at http://www.iana.org/form/media-types ? That covers mime types, but not file extensions, so it's not really what *I* thought Barry was talking about. Actually, I guess the first question is whether .py is reserved for Python scripts. If it is, then certainly /pyz should be reserved in a similar way for Python Zip Applications. If there's no formal registration of .py, then it seems a bit unreasonable to bother for .pyz... Also, I don't think reserving anything is something I, as an individual (and specifically a non-Unix user) should do. It probably should be handled by the PSF, as the process seems to need a contact email address... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 2/17/2015 1:40 PM, Barry Warsaw wrote: I agree, for deployment to environments where you can guarantee that the basic Python infrastructure is available. It may not be though, and perhaps we can relegate that to py2exe, cx_freeze, and friends, although I would say that having such support built into Python would be very cool, and of high value to our users. That's not to play down the usefulness of good executable zip support, pared of the stdlib and interpreter. OTOH, it would be nice to think ahead, where the same tool could generate both types of single-file executables. Thinking ahead? Novel idea! My dream scheme... A scheme for putting all the pure- Python modules (mostly user code) into a .pyz, and a concurrent scheme for putting all the non-pure- Python modules (maybe even Python itself, for platforms that are missing the necessary version of Python 3), into some other platform-dependent bundle. User instructions then become... try the .pyz, if it doesn't work, install the platform-dependent bundle and then it will. So one could build up a platform-dependent bundle that contains all stuff used in development of various projects, and have easy user instructions for deployment of various .pyz apps. Longer instructions for people that think they know what they are doing would list the version of Python and the versions of the various dependencies in the platform-dependent bundle, if they wish to install them manually, etc. I suppose licensing might, in some cases, conflict with making a single bundle for some modules and platforms. I'm already doing the .pyz apps, using .py extensions, but don't have the other piece in place it's harder, and I haven't acquired the skill set for putting the binary chunks together... other than unzipping to the filesystem. Glenn ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17.02.15 23:25, Barry Warsaw wrote: I'm not sure sys.getfilesystemencoding() is the right encoding, rather than sys.getdefaultencoding(), if you're talking about the encoding of the shebang line rather than the encoding of the resulting pyz filename. On POSIX sys.getfilesystemencoding() is the right encoding because the shebang is read by system loader which doesn't encode/decode, but uses a file name as raw bytes string. On Mac OS always is UTF-8, but sys.getdefaultencoding() can be ASCII. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 18:44, Barry Warsaw ba...@python.org wrote: The broader question of pex, pyzaa, etc. is an important one for Python, IMHO. Having a standard single-executable distribution story will help Python continue to compete on platforms that work beyond the distribution models we've come to think about. App models, and transactional systems such as Ubuntu Core, etc. are gaining mindshare and Python's traditional way of deploying applications doesn't really fit very well, and that may discourage developers from using Python. Whether Python itself wants to put a stake in the ground today for that is an open question, but adding support in Python for extension modules in zips (whether via save-to-disk or new operating system APIs) would be useful to explore. I happen to like pex, but what I really want is something like: $ pyvenv foo $ source foo/bin/activate $ pip install coolthing $ pyzip -o coolthing.pyz . $ deactivate $ ./coolthing.pyz This is outside the scope of PEP 441, but if anybody at Pycon wants to explore this further, I'm in. Barry, thanks for your comments. I'll pick up on the ones directly related to the PEP separately, but I'd like to add some thoughts on this. (I'm not at Pycon so dumping stuff in an email is the best I can do :-)) I'd like to see a good single-file bundled application format for Python (on Windows, there's py2exe, which is fantastic, but often I simply don't *want* to bundle Python and the stdlib with my code). The executable zip format is that solution, IMO. It's been around for ages, and has core support. There are certain things that are needed to make a compelling story, though. And it's definitely not a well known feature, which I hope is something PEP 441 will address, at least a bit. Handling of C extensions in zipfiles isn't so much an executable zip issue as a zipimport one. It would be nice to see something like this, but I would much rather it be an update to zipimport than be bolted on in whichever application does it. To my mind, having the core support the mechanisms is a much more robust approach (and avoids the possibility of the solution not being implemented cross-platform - look at py2exe which solves the extensions in a zipfile issue, but only for Windows). Also, modules still fail to be zip-safe for other reasons, such as needing data files to be real filesystem files (for example requests' certificate bundle). Adding more support for packages to be zip-safe would be fantastic, and would help in a lot more places than just the executable zip format. (Although promoting the executable zip format would help persuade projects to *use* those features rather than assuming a filesystem). Donald Stufft has a proposal on resource APIs for importlib that would help with this. As far as issues specific to the executable zip format are concerned, the main one is tool support. The zipapp module is intended as nothing more than the bare minimum. I think that more complex tools should live on PyPI, not in the core, for all the same reasons that pip was developed outside of the core - the ideal tool needs time to develop, needs a faster release cycle than core Python, etc, etc. At the moment, the only substantial tools I know of are pyzzer and pex. pyzzer is pretty good, IMO, but doesn't do much beyond the basic zip building. I tried pex, but it didn't work on Windows at the time - the problem I found is fixed, but I haven't really had the inclination to try again (there's only so many projects I can cope with being the Windows early adopter for :-)) One thing I would like to see, in the light of PEP 441, would be for pex to switch to using .pyz as its file extension, because that's what pex files are. (The documentation also refers to the support of __main__.py as a quirk in the Python importer which I find particularly irritating, as it's not a quirk, it's a deliberate feature implemented for precisely this sort of use case!!!) I agree that something that grabs all of the non-stdlib dependencies of a package and bundles them up automatically would be brilliant. I've intended to write something like that for a while now, but haven't had the time. But I certainly think your what I really want scenario above is entirely achievable. At least to the extent that the package code and its dependencies support running from a non-filesystem importer... A final thought - I wonder how easy it would be to write an unpacking zipimporter that worked like zipimport, but unpacked the zipfile to a temporary directory at the start and cleaned up on program termination? It would give the single-file benefits of zipimport, but would be guaranteed 100% compatible with filesystem imports, at a cost of worse startup/teardown performance, and extra disk space. It might make a nice fallback importer for troublesome packages. Paul ___ Python-Dev mailing list Python-Dev@python.org
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 15, 2015, at 05:15 PM, Petr Viktorin wrote: So I'd suggest `/usr/bin/env python3` for the default. I really don't want `/usr/bin/env anything`. In general, using /usr/bin/env is a fine tool for developers, but a terrible tool for deployment. It's just too easy for end users to break an application by mucking with their $PATH. In Debian/Ubuntu, we don't allow these shebangs in system scripts, and in fact the package build tools will rewrite shebangs to avoid them. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
I'm just now catching up on this thread, so hopefully these thoughts are still relevant. IIRC, the PEP has been scaled back to mostly a convenience around packing an existing directory into a .pyz file, and (on Windows) adding an association for those as executable Python zip file. To the extent that I care about Windows, +1. I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible. The broader question of pex, pyzaa, etc. is an important one for Python, IMHO. Having a standard single-executable distribution story will help Python continue to compete on platforms that work beyond the distribution models we've come to think about. App models, and transactional systems such as Ubuntu Core, etc. are gaining mindshare and Python's traditional way of deploying applications doesn't really fit very well, and that may discourage developers from using Python. Whether Python itself wants to put a stake in the ground today for that is an open question, but adding support in Python for extension modules in zips (whether via save-to-disk or new operating system APIs) would be useful to explore. I happen to like pex, but what I really want is something like: $ pyvenv foo $ source foo/bin/activate $ pip install coolthing $ pyzip -o coolthing.pyz . $ deactivate $ ./coolthing.pyz This is outside the scope of PEP 441, but if anybody at Pycon wants to explore this further, I'm in. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 16, 2015, at 04:16 PM, Paul Moore wrote: So, the options I see: 1. Stick with /usr/bin/env python 2. No shebang unless -p is specified 3. Unix users come up with a solution which is the same as the above for Windows users, but which suits them better. #2 seems to me to be the most reasonable alternative. The resulting pyz files (built w/o -p) would still be explicitly executable, just like .py files. However, -p must be able to accept any number of strings, including /usr/bin/env python3 if the user wants that. Probably the best thing to do (on *nix at least) is, if the path is absolute, use the given string verbatim. If the path is relative, search for the given executable on $PATH and use the first one found. If nothing is found, use what's given explicitly. I think that will give all the reasonable use cases. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 15, 2015, at 05:21 PM, Thomas Wouters wrote: FWIW, we have a similar tool to 'pyzaa' at Google, although a lot older (it has a pure-python implementation of zipimport, because it had to work with Python 2.2 back in the day) that *does* support extension modules and other shared libraries (by extracting them to disk on program startup.) We're also working to replace that by loading the extension modules directly from the archive (using dlopen_with_offset as described by ppluzhnikov in https://sourceware.org/bugzilla/show_bug.cgi?id=11767, although it should also be possible using dlopen_ehdr/dlopen_phdr), which requires that extension modules are stored uncompressed (simple) and page-aligned (harder, as the zipfile.ZipFile class doesn't directly support page-aligning anything, but it turns out it's easy to hack around by overriding _writecheck :P) And yes, we really are in a position to modify glibc to make our life distributing Python applications easier, internally; the old code involves shell and Python bootstrap code that tries very hard to avoid race conditions and security problems, which is a pain to maintain. So, let's add this to zipimport so we can remove an unfortunate limitation and avoid everyone having to reinvent the wheel! Oh, and the dlopen extension sound pretty cool. Can't wait until Google gets that into glibc so us mere mortals can use it. Seems reasonable to add this to zipimport too, with whatever feature guards are necessary. You'll want to give your future self an opportunity to trot out the keys to the time machine. :) It might be nice to consider those use-cases in pyzapp as well, especially once the glibc feature is released. It requires some fairly big changes to zipimport (I ended up rewriting the whole thing) but we can easily opensource the code, if there's any interest at all. +1 I'd be happy to discuss this in more detail at PyCon, at which time we should be deploying code internally using all of this. +1 Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 16, 2015, at 11:21 PM, Paul Moore wrote: Default Interpreter --- The initial draft of this PEP proposed using ``/usr/bin/env python`` as the default interpreter. The other reasonable alternative for a default shebang is sys.executable. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 18:52, Barry Warsaw ba...@python.org wrote:
So, the options I see:
1. Stick with /usr/bin/env python
2. No shebang unless -p is specified
3. Unix users come up with a solution which is the same as the above
for Windows users, but which suits them better.
#2 seems to me to be the most reasonable alternative. The resulting pyz files
(built w/o -p) would still be explicitly executable, just like .py files.
I'm pretty sure that's the way the general feeling is going.
However, -p must be able to accept any number of strings, including
/usr/bin/env python3 if the user wants that.
The code simply writes
'#!{}\n'.format(p_option).encode(sys.filesystemencoding()) to the
file, so you can put whatever you want in. Given that it isn't the
name of the Python executable, maybe the option should be
--interpreter instead?
Oh, and am I right that the shebang line should be encoded using the
filesystem encoding on Unix? I know 99.999% of use cases will be
ascii, but someone could have a Python interpreter in
/home/léon/.local/bin/python...
Probably the best thing to do (on *nix at least) is, if the path is absolute,
use the given string verbatim. If the path is relative, search for the given
executable on $PATH and use the first one found. If nothing is found, use
what's given explicitly.
I'm not quite sure what you mean here, but maybe you're thinking that
the -p option is the executable name rather than what gets put in the
#! line directly? Let me know if it's not covered by what I've already
said.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 18:56, Barry Warsaw ba...@python.org wrote: The initial draft of this PEP proposed using ``/usr/bin/env python`` as the default interpreter. The other reasonable alternative for a default shebang is sys.executable. That's instantly non-portable. On my PC, it'd give C:\Apps\Python34\python.exe. Which won't work on Unix, or on PCs that use the standard install location, or when I upgrade to Python 3.5... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 08:58 PM, Paul Moore wrote: But I would say that anything that is added to zipimport should be cross-platform. Having support for C extensions in zipimport on Unix only will just add another way in which Python applications can inadvertantly be non-portable... (It should be possible to support both Windows and Unix, though, as py2exe has the Windows side of things covered so presumably the code used in py2exe could be ported to zipimport). The actual import an extension module support in zipimport does need to be cross-platform, but it can work differently depending on the platform. For example, if extended-dlopen is available on your Linux machine, zipimport could just use that. For a traditional-dlopen-only machine, it would copy-to-filesystem. It would do whatever Windowsy thing makes sense over there too. I'm much less concerned about the cross-platform portability of the resulting pyz files. There's lots of reasons why an application built in (or for) my Linux machine might not work properly on your Windows machine, but FWIW if it were the only problem, doing a native rebuild on the target platform (or via supported cross-building) would be fine with me. Cheers, -Barry pgpD_dIRHk2Px.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 08:53 PM, Paul Moore wrote: On 17 February 2015 at 18:56, Barry Warsaw ba...@python.org wrote: The initial draft of this PEP proposed using ``/usr/bin/env python`` as the default interpreter. The other reasonable alternative for a default shebang is sys.executable. That's instantly non-portable. On my PC, it'd give C:\Apps\Python34\python.exe. Which won't work on Unix, or on PCs that use the standard install location, or when I upgrade to Python 3.5... But it doesn't matter (at least to me). The scenarios I want to support don't include building a pyz on my Linux machine, giving it to you, and having you able to run it natively on your Windows machine. There are lots of interesting use cases we could still support. I could build a single-file application and deploy it into my OS's system archives. I could deploy that thing into my platform's app store. I could hand it over to a colleague running an identical version of the OS. I could deploy it into my data center. I could install it on my phone. Don't discount sys.executable to quickly. :) Cheers, -Barry pgpmiI1N7GM6i.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 19:00, Barry Warsaw ba...@python.org wrote: It might be nice to consider those use-cases in pyzapp as well, especially once the glibc feature is released. It requires some fairly big changes to zipimport (I ended up rewriting the whole thing) but we can easily opensource the code, if there's any interest at all. +1 +1 from me also. Anything that makes it easier for packages to work under zipimport. But I would say that anything that is added to zipimport should be cross-platform. Having support for C extensions in zipimport on Unix only will just add another way in which Python applications can inadvertantly be non-portable... (It should be possible to support both Windows and Unix, though, as py2exe has the Windows side of things covered so presumably the code used in py2exe could be ported to zipimport). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 18:44, Barry Warsaw ba...@python.org wrote: I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible. I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately. But absolutely, yes, the intention is that .pyz is officially the standard extension for Python zip applications. (And for that matter, I'd like the PEP to promote the specific term Python Zip Application as the correct way to refer to these things - without a concrete term to use, it's very hard for the idea to get traction). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 02/17/2015 01:13 PM, Barry Warsaw wrote: There are lots of interesting use cases we could still support. I could build a single-file application and deploy it into my OS's system archives. I could deploy that thing into my platform's app store. I could hand it over to a colleague running an identical version of the OS. I could deploy it into my data center. I could install it on my phone. All those locations have Python installed in the same place? Even your phone?? Don't discount sys.executable too quickly. :) I suppose it could be an option. ;) -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 08:52 PM, Paul Moore wrote:
I'm pretty sure that's the way the general feeling is going.
Though the more I think about it, the more I like sys.executable. :)
However, -p must be able to accept any number of strings, including
/usr/bin/env python3 if the user wants that.
The code simply writes
'#!{}\n'.format(p_option).encode(sys.filesystemencoding()) to the file, so
you can put whatever you want in. Given that it isn't the name of the Python
executable, maybe the option should be --interpreter instead?
(Quoting out of order.)
I'm not quite sure what you mean here, but maybe you're thinking that
the -p option is the executable name rather than what gets put in the
#! line directly? Let me know if it's not covered by what I've already
said.
I was thinking it would be useful to mimic virtualenv's -p/--python option,
but I think that doesn't actually do the $PATH parsing, so maybe taking -p
verbatim is fine.
Oh, and am I right that the shebang line should be encoded using the
filesystem encoding on Unix? I know 99.999% of use cases will be
ascii, but someone could have a Python interpreter in
/home/léon/.local/bin/python...
Well, actually probably utf-8 in most cases, at least for Python 3 on *nix.
I'm not sure sys.getfilesystemencoding() is the right encoding, rather than
sys.getdefaultencoding(), if you're talking about the encoding of the shebang
line rather than the encoding of the resulting pyz filename.
Cheers,
-Barry
pgpwgqU3Yj0vR.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 08:44 PM, Paul Moore wrote: I'd like to see a good single-file bundled application format for Python (on Windows, there's py2exe, which is fantastic, but often I simply don't *want* to bundle Python and the stdlib with my code). I agree, for deployment to environments where you can guarantee that the basic Python infrastructure is available. It may not be though, and perhaps we can relegate that to py2exe, cx_freeze, and friends, although I would say that having such support built into Python would be very cool, and of high value to our users. That's not to play down the usefulness of good executable zip support, pared of the stdlib and interpreter. OTOH, it would be nice to think ahead, where the same tool could generate both types of single-file executables. Handling of C extensions in zipfiles isn't so much an executable zip issue as a zipimport one. Agreed. (Although promoting the executable zip format would help persuade projects to *use* those features rather than assuming a filesystem). Donald Stufft has a proposal on resource APIs for importlib that would help with this. Yep, which I support. :) pkg_resources.resource_filename() is the way to do that today and it abstracts away all the nastiness. Mostly agree with your other comments, though I haven't played with pyzzer and pex is quite nice, for the pared down use case, although it's not quite integrated with virtualenv (I've had some discussions with pex's author about that). OTOH, I think it *would* be useful to have this built into Python. We almost have too many choices, and may people I've talked to lately (including experienced Pythonistas) have trouble choosing the right tool. But that can come later; let's get some basic support into Python first, and a good start IMHO is executable zips (including support for extension modules) with the pkg_resources/importlib API additions. Cheers, -Barry pgp0pH78owXV6.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 21:13, Barry Warsaw ba...@python.org wrote: Don't discount sys.executable to quickly. :) I used pyzzer when it defaulted to sys.executable, and the experience was painful. That's on Windows, and Unix may be different, but let's just say I don't want to go there :-) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 21:10, Barry Warsaw ba...@python.org wrote: The actual import an extension module support in zipimport does need to be cross-platform, but it can work differently depending on the platform. For example, if extended-dlopen is available on your Linux machine, zipimport could just use that. For a traditional-dlopen-only machine, it would copy-to-filesystem. It would do whatever Windowsy thing makes sense over there too. Oh, absolutely. As long as the functionality is supported everywhere, the implementation can be whatever is needed. I'm much less concerned about the cross-platform portability of the resulting pyz files. There's lots of reasons why an application built in (or for) my Linux machine might not work properly on your Windows machine, but FWIW if it were the only problem, doing a native rebuild on the target platform (or via supported cross-building) would be fine with me. Yeah, it's not at all clear to me what people's intended use cases are for pyz files. Personally, I want them for writing scripts I can copy between my various PCs without hassle. (So single file saves me from having to make sure I have dependencies installed everywhere, but sys.executable ties me to a specific install location). Your use case seems similar (but on Unix, where there's less likelihood that Python will be installed in different places). I don't know if anyone is seeing them as a distribution format (i.e., they put their software up on their website for anyone to use). If Python zip applications got the ability to include binary extensions, they would *definitely* not be portable (we don't want to go down the route of wheel-like compatibility tags for an application file format). Currently, I'm trying to leave this aspect of the PEP open, so the PEP describes what the format is and how it works, but doesn't try to imply particular use cases. That way people can use it how they prefer, and the PEP doesn't put people off with a lot of caveats. My current draft of the documentation page for the zipapp module includes an example of setting a shebang line, with some notes on issues to consider for a portable shebang line if you propose distributing your application - but that's as far as I want to go. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 01:25 PM, Ethan Furman wrote: On 02/17/2015 01:13 PM, Barry Warsaw wrote: There are lots of interesting use cases we could still support. I could build a single-file application and deploy it into my OS's system archives. I could deploy that thing into my platform's app store. I could hand it over to a colleague running an identical version of the OS. I could deploy it into my data center. I could install it on my phone. All those locations have Python installed in the same place? Even your phone?? But of course! :) Cheers, -Barry pgpy0nxwsuCpI.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 2/17/2015 10:52 AM, Barry Warsaw wrote: Probably the best thing to do (on *nix at least) is, if the path is absolute, use the given string verbatim. If the path is relative, search for the given executable on $PATH and use the first one found. If nothing is found, use what's given explicitly. I think searching would be inappropriate. Just use what is given, as a string. If the user wants a search, they can use `which`. Otherwise, setting to a relative path reliably would be hard (and sometimes desirable). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 21:40, Barry Warsaw ba...@python.org wrote: OTOH, I think it *would* be useful to have this built into Python. We almost have too many choices, and may people I've talked to lately (including experienced Pythonistas) have trouble choosing the right tool. But that can come later; let's get some basic support into Python first, and a good start IMHO is executable zips (including support for extension modules) with the pkg_resources/importlib API additions. I see no problem with either having zipapp develop into a much fuller-featured solution, or having an externally developed tool brought into the stdlib and replacing zipapp. I just don't think there's anything mature enough for that role out there yet, and I don't feel that we know enough yet to design the right solution from scratch in the PEP. As you say, though, that can come later. If 3.5 includes PEP 441 and Donald's proposed importlib additions, I'll be happy. If it also includes zipimport support for C extensions I'll be ecstatic! (But I suspect that one's more a 3.6 timeframe) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 09:52 PM, Paul Moore wrote: I see no problem with either having zipapp develop into a much fuller-featured solution, or having an externally developed tool brought into the stdlib and replacing zipapp. I just don't think there's anything mature enough for that role out there yet, and I don't feel that we know enough yet to design the right solution from scratch in the PEP. As you say, though, that can come later. If 3.5 includes PEP 441 and Donald's proposed importlib additions, I'll be happy. If it also includes zipimport support for C extensions I'll be ecstatic! (But I suspect that one's more a 3.6 timeframe) Agreed! The only difference between executable zips and the full-featured self-contained app is the goop you put at the front of the file, right? :) Cheers, -Barry pgp8tGSl3jwSy.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 09:45 PM, Paul Moore wrote: If Python zip applications got the ability to include binary extensions, they would *definitely* not be portable (we don't want to go down the route of wheel-like compatibility tags for an application file format). Agreed! Currently, I'm trying to leave this aspect of the PEP open, so the PEP describes what the format is and how it works, but doesn't try to imply particular use cases. That way people can use it how they prefer, and the PEP doesn't put people off with a lot of caveats. My current draft of the documentation page for the zipapp module includes an example of setting a shebang line, with some notes on issues to consider for a portable shebang line if you propose distributing your application - but that's as far as I want to go. I think that's totally reasonable. Admittedly, I was hijacking your PEP discussion for some pie-in-the-sky musing. :) Cheers, -Barry pgpLu9N2fmjsg.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Sun, Feb 15, 2015 at 1:45 PM, Serhiy Storchaka storch...@gmail.com wrote: On 15.02.15 18:21, Thomas Wouters wrote: which requires that extension modules are stored uncompressed (simple) and page-aligned (harder, as the zipfile.ZipFile class doesn't directly support page-aligning anything It is possible to add this feature to ZipFile. It can be useful because will allow to mmap uncompressed files in ZIP file. One of the cool things about ZIP files is that they ignore any data that is not referenced by the index (which is at the end of the file). That is why we can put a #! at the beginning of the file, prefix .exe self-extractors, or put extra data between files in the archive. You could probably even overlap archive members if you were motivated. But it would be best to let the uncompressed extensions be standard ZIP members. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
Paul Moore wrote: On 15 February 2015 at 18:25, Petr Viktorin encu...@gmail.com wrote: On POSIXy systems the python3 symlink is created in all venvs. I thought (perhaps naïvely) that Windows doesn't do shebangs natively, so there's some Python-specific mechanism around them, which should handle python3. Windows doesn't have python2 or python3 commands, just python. To choose which version you use, set PATH or use an explicit path to the executable. The launcher offers shebang handling for scripts, and special-cases the following values for Unix compatibility: #!/usr/bin/env python #!/usr/bin/python #!/usr/local/bin/python They default to the default Python which is normally the latest Python 2 version available, but which can be configured by the user. They also support adding an explicit version (python2, python3, python2.7, python3.4, ...) As far as I know, this is *identical* behaviour to Unix - even to the incredibly annoying (to me) choice of Python 2 as a default. So I reconfigure the default in my personal settings to Python 3. Unix users can do this too (although it may involve a symlink in a ~/bin directory rather than an ini file change). We could also add special-cases for #!/usr/...python3 in the launcher on Windows. Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 02/16/2015 08:16 AM, Paul Moore wrote: The alternative, I guess, is to have *no* default and write no shebang unless -p is specified. This seems to make sense. So, the options I see: 1. Stick with /usr/bin/env python 2. No shebang unless -p is specified 3. Unix users come up with a solution which is the same as the above for Windows users, but which suits them better. +1 on 2 +0.5 on 1 -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
I don't know what anyone else does, but in cases where I have both on my windows box, I do use python2(.x) and python3(.y) . If I only have one version on the box, I use the generic name of course. (I don't often have only one version on my boxes though. 2.x inevitably gets drug in in for some reason or another and I hardly ever uninstall old versions of 3.x) I don't use the launcher though, so I might be out-of-scope entirely. (in which case, sorry for the noise) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
If I only have one version on my box, yes, I only use python. But like I said, for me personally, that situation doesn't last very long. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 18:25, Petr Viktorin encu...@gmail.com wrote: On POSIXy systems the python3 symlink is created in all venvs. I thought (perhaps naïvely) that Windows doesn't do shebangs natively, so there's some Python-specific mechanism around them, which should handle python3. Windows doesn't have python2 or python3 commands, just python. To choose which version you use, set PATH or use an explicit path to the executable. The launcher offers shebang handling for scripts, and special-cases the following values for Unix compatibility: #!/usr/bin/env python #!/usr/bin/python #!/usr/local/bin/python They default to the default Python which is normally the latest Python 2 version available, but which can be configured by the user. They also support adding an explicit version (python2, python3, python2.7, python3.4, ...) As far as I know, this is *identical* behaviour to Unix - even to the incredibly annoying (to me) choice of Python 2 as a default. So I reconfigure the default in my personal settings to Python 3. Unix users can do this too (although it may involve a symlink in a ~/bin directory rather than an ini file change). I see no reason for the zipapp module to buck the trend that the default Python is the latest 2.X version unless the user configures it differently themselves. Hence the default of #!/usr/bin/env python. Certainly that means that users who use the default have to be prepared for the fact that they are subject to user configuration changes, and yes that includes running under either of Python 2 or 3. If you don't like that, force a particular version with -p, and be prepared for the *other* issues this may cause (user doesn't have that version, for example). The alternative, I guess, is to have *no* default and write no shebang unless -p is specified. That matches how .py files work (you have to type the shebang line in if you want it) and works the same way (on Unix, a file without a shebang must be run with an explicit python command, and on Windows it uses the default Python via the file type association). Honestly, I'd be happy with this - the Windows experience is identical, and it saves choosing something inappropriate (in the face of ambiguity, refuse the temptation to guess :-)) So, the options I see: 1. Stick with /usr/bin/env python 2. No shebang unless -p is specified 3. Unix users come up with a solution which is the same as the above for Windows users, but which suits them better. I don't see changing the Windows behaviour as an option. If nobody comes up with anything for (3) I'll take the consensus between (1) or (2). If there's no consensus, I'll pick something. Probably 1, as why change if nobody has a better idea? One final thought. This issue is no different from shebang lines in .py scripts - the only real difference is that it's easy to see and edit the shebang in a .py script. It may be that what's really needed here is an option for the user to display and update the shebang line in a pyz archive. I'll look at adding that to the PEP, regardless of the conclusion to this debate. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 16 February 2015 at 16:42, Paul Moore p.f.mo...@gmail.com wrote: My point is that on Windows, users typically don't change the executable name they use[1], but rather configure the python (or py) command to do what they want. So I think that on Windows we should follow that convention and execute whatever python/py execute. One other thought. We could add short form options for -p to the zipapp command: -p X[.Y]Use #!/usr/bin/env pythonX[.Y] as the shebang. I'd definitely want the default to be python in that case, though, as it's the one you can't specify in a short form otherwise. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 16 February 2015 at 16:34, Steve Dower steve.do...@microsoft.com wrote: As far as I know, this is *identical* behaviour to Unix - even to the incredibly annoying (to me) choice of Python 2 as a default. So I reconfigure the default in my personal settings to Python 3. Unix users can do this too (although it may involve a symlink in a ~/bin directory rather than an ini file change). We could also add special-cases for #!/usr/...python3 in the launcher on Windows. The launcher handles that. It runs the same thing as py -3 runs. Which may not be the same as what py runs (in my case it runs 3.5a1 where py runs 3.4). Arguably that's an odd choice, but it's simply that I only use py in the normal course of events so I only set the python default, not the python3 default. My point is that on Windows, users typically don't change the executable name they use[1], but rather configure the python (or py) command to do what they want. So I think that on Windows we should follow that convention and execute whatever python/py execute. Paul [1] As usual with anything like this, it's hard to get a sense of what's typical so if hordes of Windows users suddenly post saying they routinely use python2 and python3 commands, I'll happily concede I'm not the norm here and ask someone to step up and document recommended practices on Windows better, and I'll update the PEP to follow them :-) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 16 February 2015 at 17:00, Mark Young marky1...@gmail.com wrote: I don't know what anyone else does, but in cases where I have both on my windows box, I do use python2(.x) and python3(.y) . If I only have one version on the box, I use the generic name of course. (I don't often have only one version on my boxes though. 2.x inevitably gets drug in in for some reason or another and I hardly ever uninstall old versions of 3.x) I don't use the launcher though, so I might be out-of-scope entirely. (in which case, sorry for the noise) No, that's good to know. One clarification, though - if you only have Python 3, I get the impression you use python and not python3 then...? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
Paul Moore wrote: The alternative, I guess, is to have *no* default and write no shebang unless -p is specified. +1. That sounds like a very good idea to me. -- Greg ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 17:46, Daniel Holth dho...@gmail.com wrote: Go ahead, make my pep. I will appreciate seeing it happen. Here is a draft update for PEP 441. It's still a work in progress - in particular I want to wait for consensus on the issue of the default interpreter before finalising it. But I thought it would be worth having a full spec available for people. PEP: 441 Title: Improving Python ZIP Application Support Version: $Revision$ Last-Modified: $Date$ Author: Daniel Holth dho...@gmail.com, Paul Moore p.f.mo...@gmail.com Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 30 March 2013 Post-History: 30 March 2013, 1 April 2013, 16 February 2015 Improving Python ZIP Application Support Python has had the ability to execute directories or ZIP-format archives as scripts since version 2.6 [1]_. When invoked with a zip file or directory as its first argument the interpreter adds that directory to sys.path and executes the __main__ module. These archives provide a great way to publish software that needs to be distributed as a single file script but is complex enough to need to be written as a collection of modules. This feature is not as popular as it should be mainly because it was not promoted as part of Python 2.6 [2]_, so that it is relatively unknown, but also because the Windows installer does not register a file extension (other than .py) for this format of file, to associate with the launcher. This PEP proposes to fix these problems by re-publicising the feature, defining the .pyz and .pyzw extensions as “Python ZIP Applications” and “Windowed Python ZIP Applications”, and providing some simple tooling to manage the format. A New Python ZIP Application Extension == The Python 3.5 installer will associate .pyz and .pyzw “Python ZIP Applications” with the platform launcher so they can be executed. A .pyz archive is a console application and a .pyzw archive is a windowed application, indicating whether the console should appear when running the app. For UNIX users, .pyz applications should typically be prefixed with a #! line pointing to the correct Python interpreter and an optional explanation:: #!/usr/bin/env python3 # Python application packed with zipapp module (binary contents of archive) However, it is always possible to execute a .pyz application by supplying the filename to the Python interpreter directly. As background, ZIP archives are defined with a footer containing relative offsets from the end of the file. They remain valid when concatenated to the end of any other file. This feature is completely standard and is how self-extracting ZIP archives and the bdist_wininst installer format work. Minimal Tooling: The zipapp Module == This PEP also proposes including a module for working with these archives. The module will contain functions for working with Python zip application archives, and a command line interface (via ``python -m zipapp``) for their creation and manipulation. Module Interface The zipapp module will provide the following functions: ``pack(target, directory, interpreter=None, main=None)`` Writes an application archive called ``target``, containing the contents of ``directory``. If ``interpreter`` is specified, it will be written to the start of the archive as a shebang line and the file will be made executable (if no interpreter is specified, the shebang line will be omitted). If the directory contains no ``__main__.py`` file, the function will construct a ``__main__.py`` which calls the function specified in the ``main`` argument (which should be in the form ``pkg.mod:fn``). It is an error to specify ``main`` if the directory contains a ``__main__.py``, or to omit ``main`` when there is no ``__main__.py`` (as that will result in an archive which has no main function and so cannot be executed). ``get_interpreter(archive)`` Returns the interpreter specified in the shebang line of the archive. If there is no shebang, the function returns None. ``set_interpreter(archive, new_archive, interpreter=None)`` Modifies the archive's shebang line to contain the specified interpreter, and writes the updated archive to new_archive. If the interpreter is None, removes the shebang line. Command Line Usage -- The zipapp module can be run with the python -m flag. The command line interface is as follows:: python -m zipapp [options] directory Create an archive from the contents of the given directory. By default, an archive will be created with the same name as the source directory, with a .pyz extension. The following options can be specified: -o archive The destination archive will have the specified name. -p interpreter The given interpreter will be written to the shebang line
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 08:14, Paul Moore p.f.mo...@gmail.com wrote: Maybe it would be better to put something on PyPI and let it develop outside the stdlib first? The only place where a .pyz file can't easily be manipulated with the stdlib zipfile module is in writing a shebang line to the start of the archive (i.e. adding prefix bytes before the start of the zipfile data). It would be nice if the ZipFile class supported this (because to do it properly you need access to the file object that the ZipFile object wraps). Would it be reasonable to add methods to the ZipFile class to read and write the prefix data? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 13:47, Nick Coghlan ncogh...@gmail.com wrote: Or we just skip the wrapper and make python -m zipapp the recommended invocation style. Adding a wrapper later is fairly easy, but removing it would be difficult. Fine with me - the wrappers are basically useless on Windows where Tools/Scripts isn't on PATH. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 08:59, Nick Coghlan ncogh...@gmail.com wrote: The other option would to cut PEP 441 way back to *just* be about standardising and registering the file associations, and recommending the use of pip to obtain the build machinery with (whether pyzaa, pyzzer or Twitter's more comprehensive pex). It would be a short PEP, but potentially still worth it for the improved visibility of the decision when folks are trying to figure out what pyz and pyzw files are later. Ok, thinking about this a little more. Getting the extension support is the key thing on Windows - at the moment, people are faced with adding their own file associations or putting binary data in a .py file, neither of which is a nice choice. Tooling is important, though - sure, you can zip the data up and put a header on, but it's fiddly. Which brings us full circle. A simple module, executable as python -m zipapp (see below re name) which exports a single function, pack() that creates the archive. If we want to provide a script to wrap the module, like pyvenv.py does for venv, I've no objection to that - presumably it would go in Tools/Scripts? If people (like me) want to experiment with a more programmatic API for building pyz files, they can do so on PyPI, and if such a thing becomes sufficiently mature we can look then at proposing it for inclusion in the stdlib, as an extension to the zipapp module. Regarding naming, I'm happy to go with zipapp if it's your preference. Presumably the wrapper in Tools/Scripts would be pyzipapp.py? So the usage would be something like python -m zipapp [options] dir_to_zip Options: -p interpreterThe interpreter to use on the shebang line (defaulting to /usr/bin/env python) -o archive_name The name of the output file (defaulting to the source directory name with a .pyz extension) If the argument has no extension, add '.pyz' -m module:function The entry point to call (written to __main__.py) Using this is an error if there is a __main__.py, and mandatory if there isn't If you want anything more complex, it's easy enough to write your own script based on zipfile, or use one of the modules on PyPI. Does this sound reasonable? If it's OK, I'll go ahead and prepare an update to the PEP and an implementation. (Steve, looks like I may be learning how to maintain the wix files after all - wish me luck :-)) If I hear no objections in the next couple of days, I'll assume everyone's OK with it and I'll prepare a PEP update and a patch. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 23:21, Paul Moore p.f.mo...@gmail.com wrote: On 15 February 2015 at 08:59, Nick Coghlan ncogh...@gmail.com wrote: The other option would to cut PEP 441 way back to *just* be about standardising and registering the file associations, and recommending the use of pip to obtain the build machinery with (whether pyzaa, pyzzer or Twitter's more comprehensive pex). It would be a short PEP, but potentially still worth it for the improved visibility of the decision when folks are trying to figure out what pyz and pyzw files are later. Ok, thinking about this a little more. Getting the extension support is the key thing on Windows - at the moment, people are faced with adding their own file associations or putting binary data in a .py file, neither of which is a nice choice. Tooling is important, though - sure, you can zip the data up and put a header on, but it's fiddly. Which brings us full circle. A simple module, executable as python -m zipapp (see below re name) which exports a single function, pack() that creates the archive. If we want to provide a script to wrap the module, like pyvenv.py does for venv, I've no objection to that - presumably it would go in Tools/Scripts? If people (like me) want to experiment with a more programmatic API for building pyz files, they can do so on PyPI, and if such a thing becomes sufficiently mature we can look then at proposing it for inclusion in the stdlib, as an extension to the zipapp module. Regarding naming, I'm happy to go with zipapp if it's your preference. Presumably the wrapper in Tools/Scripts would be pyzipapp.py? Or we just skip the wrapper and make python -m zipapp the recommended invocation style. Adding a wrapper later is fairly easy, but removing it would be difficult. So the usage would be something like python -m zipapp [options] dir_to_zip Options: -p interpreterThe interpreter to use on the shebang line (defaulting to /usr/bin/env python) -o archive_name The name of the output file (defaulting to the source directory name with a .pyz extension) If the argument has no extension, add '.pyz' -m module:function The entry point to call (written to __main__.py) Using this is an error if there is a __main__.py, and mandatory if there isn't If you want anything more complex, it's easy enough to write your own script based on zipfile, or use one of the modules on PyPI. Does this sound reasonable? If it's OK, I'll go ahead and prepare an update to the PEP and an implementation. (Steve, looks like I may be learning how to maintain the wix files after all - wish me luck :-)) If I hear no objections in the next couple of days, I'll assume everyone's OK with it and I'll prepare a PEP update and a patch. Sounds good to me. Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates it as described? It seems a bit tidier than allocating a new PEP number and rejecting PEP 441, when the revised proposal is essentially just a simplified and renamed version of your original idea. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15.02.15 10:47, Paul Moore wrote: On 15 February 2015 at 08:14, Paul Moore p.f.mo...@gmail.com wrote: Maybe it would be better to put something on PyPI and let it develop outside the stdlib first? The only place where a .pyz file can't easily be manipulated with the stdlib zipfile module is in writing a shebang line to the start of the archive (i.e. adding prefix bytes before the start of the zipfile data). It would be nice if the ZipFile class supported this (because to do it properly you need access to the file object that the ZipFile object wraps). Would it be reasonable to add methods to the ZipFile class to read and write the prefix data? But the stdlib zipfile module supports this. with open(filename, 'wb') as f: f.write(shebang) with zipfile.PyZipFile(f, 'a') as zf: ... ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 16:21, Thomas Wouters tho...@python.org wrote: It might be nice to consider those use-cases in pyzapp as well, especially once the glibc feature is released. It requires some fairly big changes to zipimport (I ended up rewriting the whole thing) but we can easily opensource the code, if there's any interest at all. I'd be happy to discuss this in more detail at PyCon, at which time we should be deploying code internally using all of this. Those cases sound more like things to consider for Python's executable zip support. The zipapp module is purely concerned with taking a bunch of stuff and making a zipfile-with-shebang python-executable zipfile out of it. Of course if Python needed certain files to be aligned a particular way in order to execute the zipfile, that *would* be something that zipapp should take into account. But I'd say let's wait till zipfile execution needs the feature before adding it to zipapp. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
Go ahead, make my pep. I will appreciate seeing it happen. On Feb 15, 2015 8:47 AM, Nick Coghlan ncogh...@gmail.com wrote: On 15 February 2015 at 23:21, Paul Moore p.f.mo...@gmail.com wrote: On 15 February 2015 at 08:59, Nick Coghlan ncogh...@gmail.com wrote: The other option would to cut PEP 441 way back to *just* be about standardising and registering the file associations, and recommending the use of pip to obtain the build machinery with (whether pyzaa, pyzzer or Twitter's more comprehensive pex). It would be a short PEP, but potentially still worth it for the improved visibility of the decision when folks are trying to figure out what pyz and pyzw files are later. Ok, thinking about this a little more. Getting the extension support is the key thing on Windows - at the moment, people are faced with adding their own file associations or putting binary data in a .py file, neither of which is a nice choice. Tooling is important, though - sure, you can zip the data up and put a header on, but it's fiddly. Which brings us full circle. A simple module, executable as python -m zipapp (see below re name) which exports a single function, pack() that creates the archive. If we want to provide a script to wrap the module, like pyvenv.py does for venv, I've no objection to that - presumably it would go in Tools/Scripts? If people (like me) want to experiment with a more programmatic API for building pyz files, they can do so on PyPI, and if such a thing becomes sufficiently mature we can look then at proposing it for inclusion in the stdlib, as an extension to the zipapp module. Regarding naming, I'm happy to go with zipapp if it's your preference. Presumably the wrapper in Tools/Scripts would be pyzipapp.py? Or we just skip the wrapper and make python -m zipapp the recommended invocation style. Adding a wrapper later is fairly easy, but removing it would be difficult. So the usage would be something like python -m zipapp [options] dir_to_zip Options: -p interpreterThe interpreter to use on the shebang line (defaulting to /usr/bin/env python) -o archive_name The name of the output file (defaulting to the source directory name with a .pyz extension) If the argument has no extension, add '.pyz' -m module:function The entry point to call (written to __main__.py) Using this is an error if there is a __main__.py, and mandatory if there isn't If you want anything more complex, it's easy enough to write your own script based on zipfile, or use one of the modules on PyPI. Does this sound reasonable? If it's OK, I'll go ahead and prepare an update to the PEP and an implementation. (Steve, looks like I may be learning how to maintain the wix files after all - wish me luck :-)) If I hear no objections in the next couple of days, I'll assume everyone's OK with it and I'll prepare a PEP update and a patch. Sounds good to me. Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates it as described? It seems a bit tidier than allocating a new PEP number and rejecting PEP 441, when the revised proposal is essentially just a simplified and renamed version of your original idea. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 16:15, Petr Viktorin encu...@gmail.com wrote: On Sun, Feb 15, 2015 at 2:21 PM, Paul Moore p.f.mo...@gmail.com wrote: So the usage would be something like python -m zipapp [options] dir_to_zip Options: -p interpreterThe interpreter to use on the shebang line (defaulting to /usr/bin/env python) On many systems this default would mean Python 2. Even if the official recommendation changes for 3.5, the status quo might linger for a few years. On the other hand, the number of distros that don't ship Python 3 is small, and the reason they're slow-moving tends to be stability and/or compliance, so they're not the target audience for zipapp. And the python3 symlink is not going away any time soon. So I'd suggest `/usr/bin/env python3` for the default. But that will fail for users who only have Python 2 installed. And it won't pick up an activated virtualenv (assuming virtualenvs on Unix don't include a python3 command, which I believe is true - it certainly won't on Windows). As a Windows user, I believe that the command python should run the version of Python that you choose to be your default. I know it's not quite that simple on Unix, and the system python command is typically Python 2, with python3 for the system Python 3 version, but I also know that tools like pyenv work like that. Equally, I've seen lots of scripts with #!/usr/bin/env python as the shebang line, and none with #!/usr/bin/env python3. That may just be my limited experience with Unix though. I don't really want #!/usr/bin/env python3 as the default shebang on Windows (at a minimum, it would do the wrong thing for my current environment). I'm open to advice from the Unix users as to how to set things up better on Unix, but there's a whole new set of problems that will arise as soon as we have different defaults on Unix and Windows, so IMO there would need to be pretty significant benefits to justify doing that. And of course it *is* only a default - users can set whatever they want. (But getting defaults right is important, so that's not to dismiss the point). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Sun, Feb 15, 2015 at 2:21 PM, Paul Moore p.f.mo...@gmail.com wrote: So the usage would be something like python -m zipapp [options] dir_to_zip Options: -p interpreterThe interpreter to use on the shebang line (defaulting to /usr/bin/env python) On many systems this default would mean Python 2. Even if the official recommendation changes for 3.5, the status quo might linger for a few years. On the other hand, the number of distros that don't ship Python 3 is small, and the reason they're slow-moving tends to be stability and/or compliance, so they're not the target audience for zipapp. And the python3 symlink is not going away any time soon. So I'd suggest `/usr/bin/env python3` for the default. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
Go ahead, make my pep. We should make a python-dev t-shirt with this on it :) Top-posted from my Windows Phone From: Daniel Holthmailto:dho...@gmail.com Sent: 2/15/2015 9:46 To: Nick Coghlanmailto:ncogh...@gmail.com Cc: Paul Mooremailto:p.f.mo...@gmail.com; Steve Dowermailto:steve.do...@microsoft.com; Python Devmailto:python-dev@python.org Subject: Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support Go ahead, make my pep. I will appreciate seeing it happen. On Feb 15, 2015 8:47 AM, Nick Coghlan ncogh...@gmail.commailto:ncogh...@gmail.com wrote: On 15 February 2015 at 23:21, Paul Moore p.f.mo...@gmail.commailto:p.f.mo...@gmail.com wrote: On 15 February 2015 at 08:59, Nick Coghlan ncogh...@gmail.commailto:ncogh...@gmail.com wrote: The other option would to cut PEP 441 way back to *just* be about standardising and registering the file associations, and recommending the use of pip to obtain the build machinery with (whether pyzaa, pyzzer or Twitter's more comprehensive pex). It would be a short PEP, but potentially still worth it for the improved visibility of the decision when folks are trying to figure out what pyz and pyzw files are later. Ok, thinking about this a little more. Getting the extension support is the key thing on Windows - at the moment, people are faced with adding their own file associations or putting binary data in a .py file, neither of which is a nice choice. Tooling is important, though - sure, you can zip the data up and put a header on, but it's fiddly. Which brings us full circle. A simple module, executable as python -m zipapp (see below re name) which exports a single function, pack() that creates the archive. If we want to provide a script to wrap the module, like pyvenv.py does for venv, I've no objection to that - presumably it would go in Tools/Scripts? If people (like me) want to experiment with a more programmatic API for building pyz files, they can do so on PyPI, and if such a thing becomes sufficiently mature we can look then at proposing it for inclusion in the stdlib, as an extension to the zipapp module. Regarding naming, I'm happy to go with zipapp if it's your preference. Presumably the wrapper in Tools/Scripts would be pyzipapp.py? Or we just skip the wrapper and make python -m zipapp the recommended invocation style. Adding a wrapper later is fairly easy, but removing it would be difficult. So the usage would be something like python -m zipapp [options] dir_to_zip Options: -p interpreterThe interpreter to use on the shebang line (defaulting to /usr/bin/env python) -o archive_name The name of the output file (defaulting to the source directory name with a .pyz extension) If the argument has no extension, add '.pyz' -m module:function The entry point to call (written to __main__.py) Using this is an error if there is a __main__.py, and mandatory if there isn't If you want anything more complex, it's easy enough to write your own script based on zipfile, or use one of the modules on PyPI. Does this sound reasonable? If it's OK, I'll go ahead and prepare an update to the PEP and an implementation. (Steve, looks like I may be learning how to maintain the wix files after all - wish me luck :-)) If I hear no objections in the next couple of days, I'll assume everyone's OK with it and I'll prepare a PEP update and a patch. Sounds good to me. Daniel, do you mind if Paul becomes a co-author on PEP 441 and updates it as described? It seems a bit tidier than allocating a new PEP number and rejecting PEP 441, when the revised proposal is essentially just a simplified and renamed version of your original idea. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.commailto:ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 16:00, Serhiy Storchaka storch...@gmail.com wrote: Would it be reasonable to add methods to the ZipFile class to read and write the prefix data? But the stdlib zipfile module supports this. with open(filename, 'wb') as f: f.write(shebang) with zipfile.PyZipFile(f, 'a') as zf: Good point. It's not quite as easy to read the prefix data. I think you can do it by reopening the file, and reading the bytes up to byte min(i.header_offset for i in zf.infolist()). You need to reopen the file, though, and that calculation doesn't work for an empty zipfile (where you need to read to the start of the central directory instead). But it's sort of possible, and it's a pretty specialised requirement anyway (I only really needed it for testing). Thanks, Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
FWIW, we have a similar tool to 'pyzaa' at Google, although a lot older (it has a pure-python implementation of zipimport, because it had to work with Python 2.2 back in the day) that *does* support extension modules and other shared libraries (by extracting them to disk on program startup.) We're also working to replace that by loading the extension modules directly from the archive (using dlopen_with_offset as described by ppluzhnikov in https://sourceware.org/bugzilla/show_bug.cgi?id=11767, although it should also be possible using dlopen_ehdr/dlopen_phdr), which requires that extension modules are stored uncompressed (simple) and page-aligned (harder, as the zipfile.ZipFile class doesn't directly support page-aligning anything, but it turns out it's easy to hack around by overriding _writecheck :P) And yes, we really are in a position to modify glibc to make our life distributing Python applications easier, internally; the old code involves shell and Python bootstrap code that tries very hard to avoid race conditions and security problems, which is a pain to maintain. It might be nice to consider those use-cases in pyzapp as well, especially once the glibc feature is released. It requires some fairly big changes to zipimport (I ended up rewriting the whole thing) but we can easily opensource the code, if there's any interest at all. I'd be happy to discuss this in more detail at PyCon, at which time we should be deploying code internally using all of this. -- Thomas Wouters tho...@python.org Hi! I'm an email virus! Think twice before sending your email to help me spread! ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 05:33, Nick Coghlan ncogh...@gmail.com wrote: I'd definitely prefer a simple procedural API like that over offering a public stateful object-oriented API, as the latter significantly increases testing complexity without offering a sufficiently compelling gain in expressiveness to justify the additional upfront test development cost and the ongoing maintenance and support burden. The main advantage of a class API is the ability to support building an archive from data in memory (or by collecting parts from different areas on the filesystem). That's something I've had a use for on a number of occasions. OTOH, if the pyz/pyzw extensions are already (or will be) registered by the installer then the only remaining feature of this PEP [1] is the archive creation app. And designing that as we go is probably the wrong way to get something in the stdlib. Maybe it would be better to put something on PyPI and let it develop outside the stdlib first? There's already at least pyzaa and pyzzer available... Paul [1] Apart from the benefit of publicising executable zip files as a means of deploying Python code by mentioning them in a What's new section :-) ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 18:14, Paul Moore p.f.mo...@gmail.com wrote: OTOH, if the pyz/pyzw extensions are already (or will be) registered by the installer then the only remaining feature of this PEP [1] is the archive creation app. And designing that as we go is probably the wrong way to get something in the stdlib. Maybe it would be better to put something on PyPI and let it develop outside the stdlib first? There's already at least pyzaa and pyzzer available... Perhaps rather than creating yet another implementation of this, we should just add some See Also links to the relevant parts of the command line usage guide and the runpy module documentation? Particularly if the modules are pip-installable and we add the cross-references to the 2.7 and 3.4 docs as well - we didn't install pip by default back when Daniel first wrote PEP 441. [1] Apart from the benefit of publicising executable zip files as a means of deploying Python code by mentioning them in a What's new section :-) That will be covered to some degree by the mention of the file associations being registered on Windows, and we can try to persuade Larry it's worth mentioning in the release announcement itself :) The other option would to cut PEP 441 way back to *just* be about standardising and registering the file associations, and recommending the use of pip to obtain the build machinery with (whether pyzaa, pyzzer or Twitter's more comprehensive pex). It would be a short PEP, but potentially still worth it for the improved visibility of the decision when folks are trying to figure out what pyz and pyzw files are later. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Sun, Feb 15, 2015 at 5:43 PM, Paul Moore p.f.mo...@gmail.com wrote: On 15 February 2015 at 16:15, Petr Viktorin encu...@gmail.com wrote: On Sun, Feb 15, 2015 at 2:21 PM, Paul Moore p.f.mo...@gmail.com wrote: So the usage would be something like python -m zipapp [options] dir_to_zip Options: -p interpreterThe interpreter to use on the shebang line (defaulting to /usr/bin/env python) On many systems this default would mean Python 2. Even if the official recommendation changes for 3.5, the status quo might linger for a few years. On the other hand, the number of distros that don't ship Python 3 is small, and the reason they're slow-moving tends to be stability and/or compliance, so they're not the target audience for zipapp. And the python3 symlink is not going away any time soon. So I'd suggest `/usr/bin/env python3` for the default. But that will fail for users who only have Python 2 installed. And it won't pick up an activated virtualenv (assuming virtualenvs on Unix don't include a python3 command, which I believe is true - it certainly won't on Windows). Yes, it would fail with only Python 2 installed. I don't think that's a problem. On POSIXy systems the python3 symlink is created in all venvs. I thought (perhaps naïvely) that Windows doesn't do shebangs natively, so there's some Python-specific mechanism around them, which should handle python3. If the app was single-source compatible with both 2 and 3, then `/usr/bin/env python` would be a good choice. But I don't think apps (as opposed to libraries) have much incentive to keep the backwards compatibility. As a Windows user, I believe that the command python should run the version of Python that you choose to be your default. I know it's not quite that simple on Unix, and the system python command is typically Python 2, with python3 for the system Python 3 version, but I also know that tools like pyenv work like that. Equally, I've seen lots of scripts with #!/usr/bin/env python as the shebang line, and none with #!/usr/bin/env python3. That may just be my limited experience with Unix though. On Linux, what version python is depends on the distro. Currently the recommendation from Python upstream is for it to be Python 2, though, and I know of just one distro that does not do this (Arch, because it switched before the recommendation was put in place). If you're in a venv, then of python is that venv's Python, of course. But the default mode of operation for Python-unaware users is to not be in a venv. I don't really want #!/usr/bin/env python3 as the default shebang on Windows (at a minimum, it would do the wrong thing for my current environment). I'm open to advice from the Unix users as to how to set things up better on Unix, but there's a whole new set of problems that will arise as soon as we have different defaults on Unix and Windows, so IMO there would need to be pretty significant benefits to justify doing that. I'm not familiar with how the shebang works on Windows, but if /usr/bin/env python3 breaks things, I find it highly unfortunate. And of course it *is* only a default - users can set whatever they want. (But getting defaults right is important, so that's not to dismiss the point). I'm afraid the point is whether it's even possible to have a shebang that does the right thing on both platforms. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15/02/2015 18:06, Steve Dower wrote: Go ahead, make my pep. We should make a python-dev t-shirt with this on it :) I'll buy one provided pp isn't too high :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15.02.15 18:21, Thomas Wouters wrote: which requires that extension modules are stored uncompressed (simple) and page-aligned (harder, as the zipfile.ZipFile class doesn't directly support page-aligning anything It is possible to add this feature to ZipFile. It can be useful because will allow to mmap uncompressed files in ZIP file. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] PEP 441 - Improving Python ZIP Application Support
I'm looking at putting together a patch for CPython to implement PEP 441. In doing so, there are a few issues that I'd like to raise with the PEP. These are all to do with the supporting app pyzaa (IIRC, Nick proposed renaming this to pyzapp, which I like, but it's not a big deal either way). 1. The PEP says that the application will warn if you try to add C extensions to the archive. In practice, this will be quite hard to do - extensions vary across platforms, and it's not only C extensions that won't work from an archive (consider a DLL with a pure-python ctypes wrapper). We could check the filenames for known extensions (maybe using importlib.machinery.EXTENSION_SUFFIXES, but that varies by platform) but I think that's going to be fragile. IMO, the costs aren't worth the benefits, and I'd like to remove this proposal and simply document that applications packed up with pyzaa need to be tested to ensure they work from a zipfile. 2. The option to compile Python source seems to me to be of limited use. It ties the archive to a particular Python version (more precisely, only one version gets the benefits of precompiled bytecode). It's also tricky to implement, particularly the option to compile an existing archive, because there standard library py_compile and compileall modules work on the filesystem, and for compiling files in an archive it would be better to build the bytecode in memory (no need to use tempfiles). I'd prefer to say that if users want to include bytecode, they can precompile on disk (using compileall) for as many versions as they want. FWIW, Daniel's standalone pyzaa program doesn't currently include either of the above features. As well as removing things, I'd like to add a programmable API for the utility. At the moment I only have a rough idea of how that would work, but I'm thinking of a PyZApp class, which can be created from a filename or an open stream (like a ZipFile object) and which has methods to set the shebang line, setting the main entry point, and adding files or directories. Does that need to go into the PEP or is it OK to review the API as part of the patch? I don't mind either way, but my normal coding process tends to be to fiddle with the API as I code it up and use it in the main app, so I'd normally leave documenting the final API till it's written... If the above seems like a reasonable plan, I'll work up a patch with the intention that it gets approved and implemented in time for 3.5. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 15 February 2015 at 00:44, Paul Moore p.f.mo...@gmail.com wrote: I'm looking at putting together a patch for CPython to implement PEP 441. In doing so, there are a few issues that I'd like to raise with the PEP. These are all to do with the supporting app pyzaa (IIRC, Nick proposed renaming this to pyzapp, which I like, but it's not a big deal either way). 1. The PEP says that the application will warn if you try to add C extensions to the archive. In practice, this will be quite hard to do - extensions vary across platforms, and it's not only C extensions that won't work from an archive (consider a DLL with a pure-python ctypes wrapper). We could check the filenames for known extensions (maybe using importlib.machinery.EXTENSION_SUFFIXES, but that varies by platform) but I think that's going to be fragile. IMO, the costs aren't worth the benefits, and I'd like to remove this proposal and simply document that applications packed up with pyzaa need to be tested to ensure they work from a zipfile. 2. The option to compile Python source seems to me to be of limited use. It ties the archive to a particular Python version (more precisely, only one version gets the benefits of precompiled bytecode). It's also tricky to implement, particularly the option to compile an existing archive, because there standard library py_compile and compileall modules work on the filesystem, and for compiling files in an archive it would be better to build the bytecode in memory (no need to use tempfiles). I'd prefer to say that if users want to include bytecode, they can precompile on disk (using compileall) for as many versions as they want. FWIW, Daniel's standalone pyzaa program doesn't currently include either of the above features. Do the simplest thing that could possibly work applies, so I'd be OK with dropping both of those features. As well as removing things, I'd like to add a programmable API for the utility. At the moment I only have a rough idea of how that would work, but I'm thinking of a PyZApp class, which can be created from a filename or an open stream (like a ZipFile object) and which has methods to set the shebang line, setting the main entry point, and adding files or directories. Does that need to go into the PEP or is it OK to review the API as part of the patch? I don't mind either way, but my normal coding process tends to be to fiddle with the API as I code it up and use it in the main app, so I'd normally leave documenting the final API till it's written... I'd suggest including the API in the PEP, which can co-evolve with the reference implementation. The following suggestions are ones I came up with last time I looked closely at the draft PEP: * use zipapp for the support module name, distinct from the command line tool name (think venv vs pyvenv) * expose a zipapp.pack programmatic interface that takes all the input pieces and builds a suitable archive I'd definitely prefer a simple procedural API like that over offering a public stateful object-oriented API, as the latter significantly increases testing complexity without offering a sufficiently compelling gain in expressiveness to justify the additional upfront test development cost and the ongoing maintenance and support burden. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
