Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17.02.15 23:25, Barry Warsaw wrote: I'm not sure sys.getfilesystemencoding() is the right encoding, rather than sys.getdefaultencoding(), if you're talking about the encoding of the shebang line rather than the encoding of the resulting pyz filename. On POSIX sys.getfilesystemencoding() is the right encoding because the shebang is read by system loader which doesn't encode/decode, but uses a file name as raw bytes string. On Mac OS always is UTF-8, but sys.getdefaultencoding() can be ASCII. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.4 RPM on AIX
Hi Zach, I think it is best to remove the .spec file if it is not supported by Python developers.It is misleading to ship unsupported file within source tree. Thanks for reply. From: Zachary Ware To: "python-dev@python.org" Cc: Blaxton Sent: Monday, February 16, 2015 11:54 PM Subject: Re: [Python-Dev] Python 3.4 RPM on AIX On Mon, Feb 16, 2015 at 10:50 PM, Blaxton wrote: > I am using the spec file that comes with Python source code which downloaded > from python.org website > source file is set on spec file to file with bz2 format while there is only > .xz and zipped are available to download. > > > I thought somehow some one from development team has wrote that .spec file > and maybe supporting it. > > This is like no support. The spec file in Misc/RPM has not been maintained in quite some time; the last change to that file that wasn't just a version bump was one [1] whose commit message was "#5776: fix mistakes in python specfile. (Nobody probably uses it anyway.)" and was committed four and a half years ago. It's not unlikely that you're the first person to use the file since then :). We should probably just remove the file, but if you manage to get it working and would like to provide a patch on bugs.python.org, it would probably be accepted. [1] https://hg.python.org/cpython/rev/ef75ecd0e1a7 -- Zach ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 2/17/2015 1:40 PM, Barry Warsaw wrote: I agree, for deployment to environments where you can guarantee that the basic Python infrastructure is available. It may not be though, and perhaps we can relegate that to py2exe, cx_freeze, and friends, although I would say that having such support built into Python would be very cool, and of high value to our users. That's not to play down the usefulness of good executable zip support, pared of the stdlib and interpreter. OTOH, it would be nice to think ahead, where the same tool could generate both types of single-file executables. Thinking ahead? Novel idea! My dream scheme... A scheme for putting all the pure- Python modules (mostly user code) into a .pyz, and a concurrent scheme for putting all the non-pure- Python modules (maybe even Python itself, for platforms that are missing the necessary version of Python 3), into some other platform-dependent bundle. User instructions then become... try the .pyz, if it doesn't work, install the platform-dependent bundle and then it will. So one could build up a platform-dependent bundle that contains all stuff used in development of various projects, and have easy user instructions for deployment of various .pyz apps. Longer instructions for people that think they know what they are doing would list the version of Python and the versions of the various dependencies in the platform-dependent bundle, if they wish to install them manually, etc. I suppose licensing might, in some cases, conflict with making a single bundle for some modules and platforms. I'm already doing the .pyz apps, using .py extensions, but don't have the other piece in place it's harder, and I haven't acquired the skill set for putting the binary chunks together... other than unzipping to the filesystem. Glenn ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Sun, Feb 15, 2015 at 1:45 PM, Serhiy Storchaka wrote: > On 15.02.15 18:21, Thomas Wouters wrote: >> >> which requires that extension modules are stored uncompressed (simple) >> and page-aligned (harder, as the zipfile.ZipFile class doesn't directly >> support page-aligning anything > > > It is possible to add this feature to ZipFile. It can be useful because will > allow to mmap uncompressed files in ZIP file. One of the cool things about ZIP files is that they ignore any data that is not referenced by the index (which is at the end of the file). That is why we can put a #! at the beginning of the file, prefix .exe self-extractors, or put extra data between files in the archive. You could probably even overlap archive members if you were motivated. But it would be best to let the uncompressed extensions be standard ZIP members. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 09:45 PM, Paul Moore wrote: >If Python zip applications got the ability to include binary extensions, they >would *definitely* not be portable (we don't want to go down the route of >wheel-like compatibility tags for an application file format). Agreed! >Currently, I'm trying to leave this aspect of the PEP open, so the PEP >describes what the format is and how it works, but doesn't try to imply >particular use cases. That way people can use it how they prefer, and the PEP >doesn't put people off with a lot of caveats. My current draft of the >documentation page for the zipapp module includes an example of setting a >shebang line, with some notes on issues to consider for a portable shebang >line if you propose distributing your application - but that's as far as I >want to go. I think that's totally reasonable. Admittedly, I was hijacking your PEP discussion for some pie-in-the-sky musing. :) Cheers, -Barry pgpLu9N2fmjsg.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 09:52 PM, Paul Moore wrote: >I see no problem with either having zipapp develop into a much >fuller-featured solution, or having an externally developed tool >brought into the stdlib and replacing zipapp. I just don't think >there's anything mature enough for that role out there yet, and I >don't feel that we know enough yet to design the right solution from >scratch in the PEP. > >As you say, though, that can come later. If 3.5 includes PEP 441 and >Donald's proposed importlib additions, I'll be happy. If it also >includes zipimport support for C extensions I'll be ecstatic! (But I >suspect that one's more a 3.6 timeframe) Agreed! The only difference between executable zips and the full-featured self-contained app is the goop you put at the front of the file, right? :) Cheers, -Barry pgp8tGSl3jwSy.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 21:40, Barry Warsaw wrote: > OTOH, I think it *would* be useful to have this built into Python. We almost > have too many choices, and may people I've talked to lately (including > experienced Pythonistas) have trouble choosing the right tool. But that can > come later; let's get some basic support into Python first, and a good start > IMHO is executable zips (including support for extension modules) with the > pkg_resources/importlib API additions. I see no problem with either having zipapp develop into a much fuller-featured solution, or having an externally developed tool brought into the stdlib and replacing zipapp. I just don't think there's anything mature enough for that role out there yet, and I don't feel that we know enough yet to design the right solution from scratch in the PEP. As you say, though, that can come later. If 3.5 includes PEP 441 and Donald's proposed importlib additions, I'll be happy. If it also includes zipimport support for C extensions I'll be ecstatic! (But I suspect that one's more a 3.6 timeframe) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 21:13, Barry Warsaw wrote: > Don't discount sys.executable to quickly. :) I used pyzzer when it defaulted to sys.executable, and the experience was painful. That's on Windows, and Unix may be different, but let's just say I don't want to go there :-) Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 21:10, Barry Warsaw wrote: > The actual "import an extension module" support in zipimport does need to be > cross-platform, but it can work differently depending on the platform. For > example, if extended-dlopen is available on your Linux machine, zipimport > could just use that. For a traditional-dlopen-only machine, it would > copy-to-filesystem. It would do whatever Windowsy thing makes sense over > there too. Oh, absolutely. As long as the functionality is supported everywhere, the implementation can be whatever is needed. > I'm much less concerned about the cross-platform portability of the resulting > pyz files. There's lots of reasons why an application built in (or for) my > Linux machine might not work properly on your Windows machine, but FWIW if it > were the only problem, doing a native rebuild on the target platform (or via > supported cross-building) would be fine with me. Yeah, it's not at all clear to me what people's intended use cases are for pyz files. Personally, I want them for writing scripts I can copy between my various PCs without hassle. (So single file saves me from having to make sure I have dependencies installed everywhere, but sys.executable ties me to a specific install location). Your use case seems similar (but on Unix, where there's less likelihood that Python will be installed in different places). I don't know if anyone is seeing them as a distribution format (i.e., they put their software up on their website for anyone to use). If Python zip applications got the ability to include binary extensions, they would *definitely* not be portable (we don't want to go down the route of wheel-like compatibility tags for an application file format). Currently, I'm trying to leave this aspect of the PEP open, so the PEP describes what the format is and how it works, but doesn't try to imply particular use cases. That way people can use it how they prefer, and the PEP doesn't put people off with a lot of caveats. My current draft of the documentation page for the zipapp module includes an example of setting a shebang line, with some notes on issues to consider for a portable shebang line if you propose distributing your application - but that's as far as I want to go. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 01:25 PM, Ethan Furman wrote: >On 02/17/2015 01:13 PM, Barry Warsaw wrote: > >> There are lots of interesting use cases we could still support. I could >> build a single-file application and deploy it into my OS's system archives. >> I could deploy that thing into my platform's app store. I could hand it >> over to a colleague running an identical version of the OS. I could deploy >> it into my data center. I could install it on my phone. > >All those locations have Python installed in the same place? Even your >phone?? But of course! :) Cheers, -Barry pgpy0nxwsuCpI.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 08:44 PM, Paul Moore wrote: >I'd like to see a good single-file bundled application format for >Python (on Windows, there's py2exe, which is fantastic, but often I >simply don't *want* to bundle Python and the stdlib with my code). I agree, for deployment to environments where you can guarantee that the basic Python infrastructure is available. It may not be though, and perhaps we can relegate that to py2exe, cx_freeze, and friends, although I would say that having such support built into Python would be very cool, and of high value to our users. That's not to play down the usefulness of good executable zip support, pared of the stdlib and interpreter. OTOH, it would be nice to think ahead, where the same tool could generate both types of single-file executables. >Handling of C extensions in zipfiles isn't so much an executable zip >issue as a zipimport one. Agreed. >(Although promoting the executable zip format would help persuade projects to >*use* those features rather than assuming a filesystem). Donald Stufft has a >proposal on resource APIs for importlib that would help with this. Yep, which I support. :) pkg_resources.resource_filename() is the way to do that today and it abstracts away all the nastiness. Mostly agree with your other comments, though I haven't played with pyzzer and pex is quite nice, for the pared down use case, although it's not quite integrated with virtualenv (I've had some discussions with pex's author about that). OTOH, I think it *would* be useful to have this built into Python. We almost have too many choices, and may people I've talked to lately (including experienced Pythonistas) have trouble choosing the right tool. But that can come later; let's get some basic support into Python first, and a good start IMHO is executable zips (including support for extension modules) with the pkg_resources/importlib API additions. Cheers, -Barry pgp0pH78owXV6.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 2/17/2015 10:52 AM, Barry Warsaw wrote: Probably the best thing to do (on *nix at least) is, if the path is absolute, use the given string verbatim. If the path is relative, search for the given executable on $PATH and use the first one found. If nothing is found, use what's given explicitly. I think searching would be inappropriate. Just use what is given, as a string. If the user wants a search, they can use `which`. Otherwise, setting to a relative path reliably would be hard (and sometimes desirable). ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 08:52 PM, Paul Moore wrote:
>I'm pretty sure that's the way the general feeling is going.
Though the more I think about it, the more I like sys.executable. :)
>> However, -p must be able to accept any number of strings, including
>> "/usr/bin/env python3" if the user wants that.
>
>The code simply writes
>'#!{}\n'.format(p_option).encode(sys.filesystemencoding()) to the file, so
>you can put whatever you want in. Given that it isn't the name of the Python
>executable, maybe the option should be --interpreter instead?
(Quoting out of order.)
>I'm not quite sure what you mean here, but maybe you're thinking that
>the -p option is the executable name rather than what gets put in the
>#! line directly? Let me know if it's not covered by what I've already
>said.
I was thinking it would be useful to mimic virtualenv's -p/--python option,
but I think that doesn't actually do the $PATH parsing, so maybe taking -p
verbatim is fine.
>Oh, and am I right that the shebang line should be encoded using the
>filesystem encoding on Unix? I know 99.999% of use cases will be
>ascii, but someone could have a Python interpreter in
>/home/léon/.local/bin/python...
Well, actually probably utf-8 in most cases, at least for Python 3 on *nix.
I'm not sure sys.getfilesystemencoding() is the right encoding, rather than
sys.getdefaultencoding(), if you're talking about the encoding of the shebang
line rather than the encoding of the resulting pyz filename.
Cheers,
-Barry
pgpwgqU3Yj0vR.pgp
Description: OpenPGP digital signature
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 02/17/2015 01:13 PM, Barry Warsaw wrote: > There are lots of interesting use cases we could still support. I could build > a single-file application and deploy it into my OS's system archives. I could > deploy that thing into my platform's app store. I could hand it over to a > colleague running an identical version of the OS. I could deploy it into my > data center. I could install it on my phone. All those locations have Python installed in the same place? Even your phone?? > Don't discount sys.executable too quickly. :) I suppose it could be an option. ;) -- ~Ethan~ signature.asc Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 18:44, Barry Warsaw wrote: > I don't know exactly what the procedure would be to claim .pyz for *nix, > e.g. updating /etc/mime.types, but I think the PEP should at least mention > this. I think we want to get as official support for .pyz files on *nix as > possible. I'll add a note to the PEP, but I have no idea how we would even go about that, so that's all I can do, unfortunately. But absolutely, yes, the intention is that ".pyz" is officially the standard extension for Python zip applications. (And for that matter, I'd like the PEP to promote the specific term "Python Zip Application" as the correct way to refer to these things - without a concrete term to use, it's very hard for the idea to get traction). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 08:53 PM, Paul Moore wrote: >On 17 February 2015 at 18:56, Barry Warsaw wrote: >>>The initial draft of this PEP proposed using ``/usr/bin/env python`` >>>as the default interpreter. >> >> The other reasonable alternative for a default shebang is sys.executable. > >That's instantly non-portable. On my PC, it'd give >C:\Apps\Python34\python.exe. Which won't work on Unix, or on PCs that >use the standard install location, or when I upgrade to Python 3.5... But it doesn't matter (at least to me). The scenarios I want to support don't include building a pyz on my Linux machine, giving it to you, and having you able to run it natively on your Windows machine. There are lots of interesting use cases we could still support. I could build a single-file application and deploy it into my OS's system archives. I could deploy that thing into my platform's app store. I could hand it over to a colleague running an identical version of the OS. I could deploy it into my data center. I could install it on my phone. Don't discount sys.executable to quickly. :) Cheers, -Barry pgpmiI1N7GM6i.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 17, 2015, at 08:58 PM, Paul Moore wrote: >But I would say that anything that is added to zipimport should be >cross-platform. Having support for C extensions in zipimport on Unix >only will just add another way in which Python applications can >inadvertantly be non-portable... (It should be possible to support >both Windows and Unix, though, as py2exe has the Windows side of >things covered so presumably the code used in py2exe could be ported >to zipimport). The actual "import an extension module" support in zipimport does need to be cross-platform, but it can work differently depending on the platform. For example, if extended-dlopen is available on your Linux machine, zipimport could just use that. For a traditional-dlopen-only machine, it would copy-to-filesystem. It would do whatever Windowsy thing makes sense over there too. I'm much less concerned about the cross-platform portability of the resulting pyz files. There's lots of reasons why an application built in (or for) my Linux machine might not work properly on your Windows machine, but FWIW if it were the only problem, doing a native rebuild on the target platform (or via supported cross-building) would be fine with me. Cheers, -Barry pgpD_dIRHk2Px.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 19:00, Barry Warsaw wrote: >>It might be nice to consider those use-cases in pyzapp as well, especially >>once the glibc feature is released. It requires some fairly big changes to >>zipimport (I ended up rewriting the whole thing) but we can easily >>opensource the code, if there's any interest at all. > > +1 +1 from me also. Anything that makes it easier for packages to work under zipimport. But I would say that anything that is added to zipimport should be cross-platform. Having support for C extensions in zipimport on Unix only will just add another way in which Python applications can inadvertantly be non-portable... (It should be possible to support both Windows and Unix, though, as py2exe has the Windows side of things covered so presumably the code used in py2exe could be ported to zipimport). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 18:56, Barry Warsaw wrote: >>The initial draft of this PEP proposed using ``/usr/bin/env python`` >>as the default interpreter. > > The other reasonable alternative for a default shebang is sys.executable. That's instantly non-portable. On my PC, it'd give C:\Apps\Python34\python.exe. Which won't work on Unix, or on PCs that use the standard install location, or when I upgrade to Python 3.5... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 18:52, Barry Warsaw wrote:
>>So, the options I see:
>>
>>1. Stick with /usr/bin/env python
>>2. No shebang unless -p is specified
>>3. Unix users come up with a solution which is the same as the above
>>for Windows users, but which suits them better.
>
> #2 seems to me to be the most reasonable alternative. The resulting pyz files
> (built w/o -p) would still be explicitly executable, just like .py files.
I'm pretty sure that's the way the general feeling is going.
> However, -p must be able to accept any number of strings, including
> "/usr/bin/env python3" if the user wants that.
The code simply writes
'#!{}\n'.format(p_option).encode(sys.filesystemencoding()) to the
file, so you can put whatever you want in. Given that it isn't the
name of the Python executable, maybe the option should be
--interpreter instead?
Oh, and am I right that the shebang line should be encoded using the
filesystem encoding on Unix? I know 99.999% of use cases will be
ascii, but someone could have a Python interpreter in
/home/léon/.local/bin/python...
> Probably the best thing to do (on *nix at least) is, if the path is absolute,
> use the given string verbatim. If the path is relative, search for the given
> executable on $PATH and use the first one found. If nothing is found, use
> what's given explicitly.
I'm not quite sure what you mean here, but maybe you're thinking that
the -p option is the executable name rather than what gets put in the
#! line directly? Let me know if it's not covered by what I've already
said.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On 17 February 2015 at 18:44, Barry Warsaw wrote: > The broader question of pex, pyzaa, etc. is an important one for Python, > IMHO. Having a standard single-executable distribution story will help Python > continue to compete on platforms that work beyond the distribution models > we've come to think about. App models, and transactional systems such as > Ubuntu Core, etc. are gaining mindshare and Python's traditional way of > deploying applications doesn't really fit very well, and that may discourage > developers from using Python. > > Whether Python itself wants to put a stake in the ground today for that is an > open question, but adding support in Python for extension modules in zips > (whether via save-to-disk or new operating system APIs) would be useful to > explore. > > I happen to like pex, but what I really want is something like: > > $ pyvenv foo > $ source foo/bin/activate > $ pip install coolthing > $ pyzip -o coolthing.pyz . > $ deactivate > $ ./coolthing.pyz > > This is outside the scope of PEP 441, but if anybody at Pycon wants to explore > this further, I'm in. Barry, thanks for your comments. I'll pick up on the ones directly related to the PEP separately, but I'd like to add some thoughts on this. (I'm not at Pycon so dumping stuff in an email is the best I can do :-)) I'd like to see a good single-file bundled application format for Python (on Windows, there's py2exe, which is fantastic, but often I simply don't *want* to bundle Python and the stdlib with my code). The executable zip format is that solution, IMO. It's been around for ages, and has core support. There are certain things that are needed to make a compelling story, though. And it's definitely not a well known feature, which I hope is something PEP 441 will address, at least a bit. Handling of C extensions in zipfiles isn't so much an executable zip issue as a zipimport one. It would be nice to see something like this, but I would much rather it be an update to zipimport than be bolted on in whichever application does it. To my mind, having the core support the mechanisms is a much more robust approach (and avoids the possibility of the solution not being implemented cross-platform - look at py2exe which solves the extensions in a zipfile issue, but only for Windows). Also, modules still fail to be zip-safe for other reasons, such as needing data files to be real filesystem files (for example requests' certificate bundle). Adding more support for packages to be zip-safe would be fantastic, and would help in a lot more places than just the executable zip format. (Although promoting the executable zip format would help persuade projects to *use* those features rather than assuming a filesystem). Donald Stufft has a proposal on resource APIs for importlib that would help with this. As far as issues specific to the executable zip format are concerned, the main one is tool support. The zipapp module is intended as nothing more than the bare minimum. I think that more complex tools should live on PyPI, not in the core, for all the same reasons that pip was developed outside of the core - the ideal tool needs time to develop, needs a faster release cycle than core Python, etc, etc. At the moment, the only substantial tools I know of are pyzzer and pex. pyzzer is pretty good, IMO, but doesn't do much beyond the basic zip building. I tried pex, but it didn't work on Windows at the time - the problem I found is fixed, but I haven't really had the inclination to try again (there's only so many projects I can cope with being the Windows early adopter for :-)) One thing I would like to see, in the light of PEP 441, would be for pex to switch to using ".pyz" as its file extension, because that's what pex files are. (The documentation also refers to the support of __main__.py as "a quirk in the Python importer" which I find particularly irritating, as it's not a quirk, it's a deliberate feature implemented for precisely this sort of use case!!!) I agree that something that grabs all of the non-stdlib dependencies of a package and bundles them up automatically would be brilliant. I've intended to write something like that for a while now, but haven't had the time. But I certainly think your "what I really want" scenario above is entirely achievable. At least to the extent that the package code and its dependencies support running from a non-filesystem importer... A final thought - I wonder how easy it would be to write an "unpacking zipimporter" that worked like zipimport, but unpacked the zipfile to a temporary directory at the start and cleaned up on program termination? It would give the single-file benefits of zipimport, but would be guaranteed 100% compatible with filesystem imports, at a cost of worse startup/teardown performance, and extra disk space. It might make a nice fallback importer for troublesome packages. Paul ___ Python-Dev mailing list Python-Dev@python.org https:
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 16, 2015, at 11:21 PM, Paul Moore wrote: >Default Interpreter >--- > >The initial draft of this PEP proposed using ``/usr/bin/env python`` >as the default interpreter. The other reasonable alternative for a default shebang is sys.executable. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 15, 2015, at 05:21 PM, Thomas Wouters wrote: >FWIW, we have a similar tool to 'pyzaa' at Google, although a lot older (it >has a pure-python implementation of zipimport, because it had to work with >Python 2.2 back in the day) that *does* support extension modules and other >shared libraries (by extracting them to disk on program startup.) We're >also working to replace that by loading the extension modules directly from >the archive (using dlopen_with_offset as described by ppluzhnikov in >https://sourceware.org/bugzilla/show_bug.cgi?id=11767, although it should >also be possible using dlopen_ehdr/dlopen_phdr), which requires that >extension modules are stored uncompressed (simple) and page-aligned >(harder, as the zipfile.ZipFile class doesn't directly support >page-aligning anything, but it turns out it's easy to hack around by >overriding _writecheck :P) And yes, we really are in a position to modify >glibc to make our life distributing Python applications easier, internally; >the old code involves shell and Python bootstrap code that tries very hard >to avoid race conditions and security problems, which is a pain to maintain. So, let's add this to zipimport so we can remove an unfortunate limitation and avoid everyone having to reinvent the wheel! Oh, and the dlopen extension sound pretty cool. Can't wait until Google gets that into glibc so us mere mortals can use it. Seems reasonable to add this to zipimport too, with whatever feature guards are necessary. You'll want to give your future self an opportunity to trot out the keys to the time machine. :) >It might be nice to consider those use-cases in pyzapp as well, especially >once the glibc feature is released. It requires some fairly big changes to >zipimport (I ended up rewriting the whole thing) but we can easily >opensource the code, if there's any interest at all. +1 >I'd be happy to discuss this in more detail at PyCon, at which time we should >be deploying code internally using all of this. +1 Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 16, 2015, at 04:16 PM, Paul Moore wrote: >So, the options I see: > >1. Stick with /usr/bin/env python >2. No shebang unless -p is specified >3. Unix users come up with a solution which is the same as the above >for Windows users, but which suits them better. #2 seems to me to be the most reasonable alternative. The resulting pyz files (built w/o -p) would still be explicitly executable, just like .py files. However, -p must be able to accept any number of strings, including "/usr/bin/env python3" if the user wants that. Probably the best thing to do (on *nix at least) is, if the path is absolute, use the given string verbatim. If the path is relative, search for the given executable on $PATH and use the first one found. If nothing is found, use what's given explicitly. I think that will give all the reasonable use cases. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
On Feb 15, 2015, at 05:15 PM, Petr Viktorin wrote: >So I'd suggest `/usr/bin/env python3` for the default. I really don't want `/usr/bin/env anything`. In general, using /usr/bin/env is a fine tool for developers, but a terrible tool for deployment. It's just too easy for end users to break an application by mucking with their $PATH. In Debian/Ubuntu, we don't allow these shebangs in system scripts, and in fact the package build tools will rewrite shebangs to avoid them. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 441 - Improving Python ZIP Application Support
I'm just now catching up on this thread, so hopefully these thoughts are still relevant. IIRC, the PEP has been scaled back to mostly a convenience around packing an existing directory into a .pyz file, and (on Windows) adding an association for those as executable Python zip file. To the extent that I care about Windows, +1. I don't know exactly what the procedure would be to claim .pyz for *nix, e.g. updating /etc/mime.types, but I think the PEP should at least mention this. I think we want to get as official support for .pyz files on *nix as possible. The broader question of pex, pyzaa, etc. is an important one for Python, IMHO. Having a standard single-executable distribution story will help Python continue to compete on platforms that work beyond the distribution models we've come to think about. App models, and transactional systems such as Ubuntu Core, etc. are gaining mindshare and Python's traditional way of deploying applications doesn't really fit very well, and that may discourage developers from using Python. Whether Python itself wants to put a stake in the ground today for that is an open question, but adding support in Python for extension modules in zips (whether via save-to-disk or new operating system APIs) would be useful to explore. I happen to like pex, but what I really want is something like: $ pyvenv foo $ source foo/bin/activate $ pip install coolthing $ pyzip -o coolthing.pyz . $ deactivate $ ./coolthing.pyz This is outside the scope of PEP 441, but if anybody at Pycon wants to explore this further, I'm in. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.4 RPM on AIX
- Original Message - > Blaxton writes: > > > I am using the spec file that comes with Python source code which > > downloaded from python.org website > > Ah, sorry, I didn't realize that. My advice still stands; while there > are Red Hat/Fedora/other-RPM-based distro workers here, I don't know > if any of them actually have anything to do with RPM-building. And > I've been following this list for over 10 years -- this is the first > I've heard of problems with the .spec in the Python sources, so I > suspect Zach is right: you're the first person to use the thing in > half a decade. The Python project itself doesn't distribute RPMs as > far as I can see. Yes, there are people here who do everything around distro RPM-building here :) AFAIK no RPM based distribution uses the specfile from the Python tarball. Our specfiles (Fedora/Red Hat Enterprise Linux) are much more complex to satisfy all the downstream policies and such. In case you're interested, you can have a look at our current python3.spec [1]. Feel free to reuse it (all Fedora specfiles are licensed under MIT license unless noted otherwise). > > This is like no support. > > Unfortunately, yes. I know I don't have the knowledge to help fix the > .spec file, and I don't even know who to tell you to ask in the project. I can help with RPM related questions (it'd probably be best to ask them privately, since their offtopic for this list), but I have pretty much zero knowledge of AIX - so I won't be able to help you with AIX-specific issues. -- Regards, Slavek Kabrda [1] http://pkgs.fedoraproject.org/cgit/python3.git/tree/python3.spec ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python 3.4 RPM on AIX
Blaxton writes: > I am using the spec file that comes with Python source code which > downloaded from python.org website Ah, sorry, I didn't realize that. My advice still stands; while there are Red Hat/Fedora/other-RPM-based distro workers here, I don't know if any of them actually have anything to do with RPM-building. And I've been following this list for over 10 years -- this is the first I've heard of problems with the .spec in the Python sources, so I suspect Zach is right: you're the first person to use the thing in half a decade. The Python project itself doesn't distribute RPMs as far as I can see. > This is like no support. Unfortunately, yes. I know I don't have the knowledge to help fix the .spec file, and I don't even know who to tell you to ask in the project. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
