Re: [Python-Dev] Packaging and binary distributions for Python 3.3
2011/10/10 PJ Eby : > On Sun, Oct 9, 2011 at 4:14 PM, Paul Moore wrote: >> >> As regards the format, bdist_dumb is about the right level - but >> having just checked it has some problems (which if I recall, have been >> known for some time, and are why bdist_dumb doesn't get used). >> Specifically, bdist_dumb puts the location of site-packages ON THE >> BUILD SYSTEM into the archive, making it useless for direct unzipping >> on a target system which has Python installed somewhere else. > > I don't know about the case for packaging/distutils2, but I know that in > original distutils, you can work around this by making bdist_dumb call the > install commands with different arguments. That is, it's a relatively > shallow flaw in bdist_dumb. Agreed. > bdist_wininst, for example, is basically a zipped bdist_dumb with altered > install arguments and an .exe header tacked on the front. (Along with a > little extra data crammed in between the two.) I'd propose that the install arguments used in bdist_wininst be transferred to bdist_dumb (or a new command bdist_binary created based on the same), because the bdist_wininst zip format has the following advantages: 1. Proven format, so it should deal with any edge cases like header files reasonably. And the code already exists. 2. Easily recognisable directory names (PLATLIB, PURELIB, etc) making detection easy without needing extra metadata. 3. At a pinch, a bdist_wininst installer could be treated as a dumb distribution without change (assuming the stdlib zip handling correctly ignores prepended data like the exe header). Then pysetup could be enhanced to recognise and install the binary format in pretty much the same way as it does source formats (add another install_method to _run_install_from_dir that copies the files to the target locations along with appropriate checking and/or metadata handling). There might be a small amount of extra work to do, to check binary version compatibility, but that shouldn't be hard. If this is useful, I could look at creating a patch. (Once I get my build environment fixed so I can get 3.3 up and running - it looks like Python 3.3 can't be built with Visual C++ Express these days, the IDE can't convert the solution files because Express Edition doesn't support 64-bit. I'll have to fish out the full version and install that...) Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On Mon, Oct 10, 2011 at 7:49 PM, Paul Moore wrote: > I'd propose that the install arguments used in bdist_wininst be > transferred to bdist_dumb (or a new command bdist_binary created based > on the same) bdist_zip, bdist_archive, bdist_simple would all work (bdist_binary is redundant, given what the 'b' stands for). The 'bdist_dumb' name has always irritated me, since the connotations more strongly favour 'stupid' than they do 'simple' (of course, a legitimate argument can be made that the default behaviour of bdist_dumb *is* pretty stupid). > If this is useful, I could look at creating a patch. (Once I get my > build environment fixed so I can get 3.3 up and running - it looks > like Python 3.3 can't be built with Visual C++ Express these days, the > IDE can't convert the solution files because Express Edition doesn't > support 64-bit. I'll have to fish out the full version and install > that...) IIRC, even the Express edition should still work once the 64 bit Platform SDK is installed. Regardless, the intent is that it should be possible to build Python with only the free tools from MS. If they broke the Express editions such that extra tools are needed, suggested updates to the devguide [1] would be appreciated. [1] http://docs.python.org/devguide/setup.html#windows Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On 10 October 2011 12:47, Nick Coghlan wrote: > IIRC, even the Express edition should still work once the 64 bit > Platform SDK is installed. Regardless, the intent is that it should be > possible to build Python with only the free tools from MS. If they > broke the Express editions such that extra tools are needed, suggested > updates to the devguide [1] would be appreciated. > > [1] http://docs.python.org/devguide/setup.html#windows It's the "once the 64 bit platform SDK is installed" bit that's the pain. To just build a test setup for a 32-bit PC it's a shame that you need to install a load of 64-bit tools. Hmm, looking at the devguide it says you should use VS 2008. If that's still the case, ignore me - I was using VS 2010 as that's what's (easily) downloadable. I've now installed VS Pro 2010. We'll see how that goes. I'd rather avoid downgrading to VS2008 (or having both at once) just for personal builds. But will if I have to. Nothing to see here (other than me not reading the docs), move right along... :-) Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On 10/10/2011 12:58, Paul Moore wrote: I've now installed VS Pro 2010. We'll see how that goes. I'd rather avoid downgrading to VS2008 (or having both at once) just for personal builds. But will if I have to. Fairly sure VS2010 won't work, Paul. At least it didn't when I was in the same situation a few months ago. I've had close to zero time to spend on Python lately so I could be wrong but I remember that I had to "downgrade" to VS2008 for that reason. TJG ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
Giampaolo Rodolà gmail.com> writes: > This let us use different stdlib features which appeared in latest > Python versions (including 3.2) throughout all our code base. > Now, what I have in mind is to release this as a public module so that > everyone who cannot upgrade to a recent python version can benefit of > newer features. There's also the logutils project, which aims to bring logging features added in recent Pythons to older Pythons - especially dictionary-based configuration and queue handlers to facilitate working with multiprocessing, ZeroMQ etc. http://code.google.com/p/logutils/ Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
On Mon, Oct 10, 2011 at 11:47 PM, Vinay Sajip wrote: > Giampaolo Rodolà gmail.com> writes: > >> This let us use different stdlib features which appeared in latest >> Python versions (including 3.2) throughout all our code base. >> Now, what I have in mind is to release this as a public module so that >> everyone who cannot upgrade to a recent python version can benefit of >> newer features. > > There's also the logutils project, which aims to bring logging features added > in > recent Pythons to older Pythons - especially dictionary-based configuration > and > queue handlers to facilitate working with multiprocessing, ZeroMQ etc. > > http://code.google.com/p/logutils/ Should we create an informational PEP or other resource to point people towards some of these forwards compatibility options? Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
Paul Moore gmail.com> writes: > I'd propose that the install arguments used in bdist_wininst be > transferred to bdist_dumb (or a new command bdist_binary created based > on the same), because the bdist_wininst zip format has the following > advantages: > > 1. Proven format, so it should deal with any edge cases like header > files reasonably. And the code already exists. > 2. Easily recognisable directory names (PLATLIB, PURELIB, etc) making > detection easy without needing extra metadata. > 3. At a pinch, a bdist_wininst installer could be treated as a dumb > distribution without change (assuming the stdlib zip handling > correctly ignores prepended data like the exe header). > > Then pysetup could be enhanced to recognise and install the binary > format in pretty much the same way as it does source formats (add > another install_method to _run_install_from_dir that copies the files > to the target locations along with appropriate checking and/or > metadata handling). A simple change to packaging will allow an archive containing a setup.cfg-based directory to be installed in the same way as a source directory. AFAICT this gives a more useful result than bdist_wininst (as you typically want to install in more places than PURELIB and PLATLIB, and the setup.cfg scheme allows for writing files to locations such as Powershell script directories for a user). > There might be a small amount of extra work to do, to check binary > version compatibility, but that shouldn't be hard. > > If this is useful, I could look at creating a patch. (Once I get my > build environment fixed so I can get 3.3 up and running - it looks > like Python 3.3 can't be built with Visual C++ Express these days, the > IDE can't convert the solution files because Express Edition doesn't > support 64-bit. I'll have to fish out the full version and install > that...) There's one thing that you touched on in an earlier post, which hasn't been further discussed: support for virtual environments. The executable installer format covers two things: packaging of version specific/compiled code, and the simplicity of point-and-click installation. This latter convenience is worth having, but the current installer stub (wininst-x.y.exe) does not know anything about virtual environments. If we care about virtual environment support (and I think we should), wininst.exe could be enhanced to provide a "Browse..." button to allow a user to select a virtual environment to install into, in addition to the detection of installed Pythons from the registry. If this is coupled with the ability to invoke a setup.cfg-based installation when the appended archive is for a setup.cfg-based directory tree, won't this pretty much tick all the boxes? Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
Nick Coghlan gmail.com> writes: > Should we create an informational PEP or other resource to point > people towards some of these forwards compatibility options? Or perhaps a page on www.python.org which is referenced by e.g. a footnote in PEP 387 (Backwards Compatibility Policy)? Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On 10 October 2011 17:12, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> I'd propose that the install arguments used in bdist_wininst be >> transferred to bdist_dumb (or a new command bdist_binary created based >> on the same), because the bdist_wininst zip format has the following >> advantages: >> >> 1. Proven format, so it should deal with any edge cases like header >> files reasonably. And the code already exists. >> 2. Easily recognisable directory names (PLATLIB, PURELIB, etc) making >> detection easy without needing extra metadata. >> 3. At a pinch, a bdist_wininst installer could be treated as a dumb >> distribution without change (assuming the stdlib zip handling >> correctly ignores prepended data like the exe header). >> >> Then pysetup could be enhanced to recognise and install the binary >> format in pretty much the same way as it does source formats (add >> another install_method to _run_install_from_dir that copies the files >> to the target locations along with appropriate checking and/or >> metadata handling). > > A simple change to packaging will allow an archive containing a > setup.cfg-based > directory to be installed in the same way as a source directory. AFAICT this > gives a more useful result than bdist_wininst (as you typically want to > install > in more places than PURELIB and PLATLIB, and the setup.cfg scheme allows for > writing files to locations such as Powershell script directories for a user). I'm not sure what you mean by a "setup.cfg-based directory". Could you clarify, and maybe explain how you'd expect to create such an archive? We may be talking at cross-purposes here. >> There might be a small amount of extra work to do, to check binary >> version compatibility, but that shouldn't be hard. >> >> If this is useful, I could look at creating a patch. (Once I get my >> build environment fixed so I can get 3.3 up and running - it looks >> like Python 3.3 can't be built with Visual C++ Express these days, the >> IDE can't convert the solution files because Express Edition doesn't >> support 64-bit. I'll have to fish out the full version and install >> that...) > > There's one thing that you touched on in an earlier post, which hasn't been > further discussed: support for virtual environments. The executable installer > format covers two things: packaging of version specific/compiled code, and the > simplicity of point-and-click installation. This latter convenience is worth > having, but the current installer stub (wininst-x.y.exe) does not know > anything > about virtual environments. If we care about virtual environment support (and > I > think we should), wininst.exe could be enhanced to provide a "Browse..." > button > to allow a user to select a virtual environment to install into, in addition > to > the detection of installed Pythons from the registry. If this is coupled with > the ability to invoke a setup.cfg-based installation when the appended archive > is for a setup.cfg-based directory tree, won't this pretty much tick all the > boxes? Agreed - but I'm looking at a pysetup install approach to work for source and binary packages, essentially replacing the use of bdist_wininst and bdist_msi with sdist/bdist_simple archives. That's a change of heart for me, as I used to argue for wininst/msi over setuptools and similar - but pysetup includes all the listing and uninstalling features I wanted, so the "one unified approach" has won me over in preference to the platform integration. Ideally bdist_wininst and bdist_msi would also integrate with pysetup and with virtual environments, but I imagine that could be pretty hard to make work cleanly, as Windows doesn't really support multiple installations of a software package... (Plus, I've no real idea about how bdist_wininst works, so while you may be right, I wouldn't know how to do anything with your suggestion :-)) Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
Thanks everybody for your feedback. I created a gcode project here: http://code.google.com/p/pycompat/ 2011/10/8 Antoine Pitrou : > There's also some stuff there that is coded in C, or that will rely on > some functionality of the core interpreter that is not easily > emulated on previous versions. But I suppose you'll find that out by > yourself. Yep, I'm still not sure what to do about this. I guess I'll just ignore that stuff in all those cases where rewriting it in python is too much effort. Toshio Kuratomi wrote: > I have a need to support a small amount of code as far back as python-2.3 > I don't suppose you're interested in that as well? ;-) I'm still not sure; 2.3 version is way too old (it doesn't even have decorators). It might make sense only in case the lib gets widely used, which I doubt. Personally, at some point I deliberately dropped support for 2.3 from all of my code/lib, mainly because I couldn't use decorators. so I don't have a real need to do this. 2011/10/9 Éric Araujo : > The issues I foresee with your lib are more technical: First, it looks > like a big bag of backported modules, classes and functions without > defined criterion for inclusion (“cool new stuff”?). I'd say the criterion for inclusion is putting in everything which can be (re)written in python 2.4, such as any, all, collections.defaultdict and property setters/deleters (2.6). Pretty much all the stuff written in C would be left out, maybe with the exception of functools module which is important (for me at least), in which case I might try to rewrite it in pure Python. I'm sharing your same doubts though. Maybe this isn't worth the effort in the first place. I'll try to write some more code and see whether this is a good candidate for a "public module". If not I'll just get back to use it as an internal "private" module. 2011/10/9 Éric Araujo : > keep on lumping new things until Python 3.4? 3.8? Won’t that become > unmanageable (boring/huge/hard)? I don't think it makes sense to go over than 3.2 version. Folks which are forced to use python 2.4 are already avoing to use 2.6 and 2.7 features, let alone 3.X only features. Plus, python 3.2 was already the latest 3.X version which still had something in common with 2.7. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
> I'm not sure what you mean by a "setup.cfg-based directory". Could > you > clarify, and maybe explain how you'd expect to create such an archive? > We may be talking at cross-purposes here. Here's how I see it: at present, you can install a project by specifying pysetup3 install path-to-directory where the named directory contains a setup.cfg (replacing setup.py) and a bunch of things to install. Exactly what to install where is specified in the setup.cfg: it covers not only python packages and modules but also arbitrary binary files. The setup.cfg format is extensible enough to allow specifying where files are installed not only declaratively (as static paths in setup.cfg) but also according to criteria computed at installation time (e.g., write some PowerShell scripts to the installing user's PowerShell profile location). Of course, since you can install arbitrary data (and record what was installed where, to allow uninstallation to work), you can certainly install DLLs too (modulo the user having write permissions for the installation location, but that's true for data files, too). In theory, therefore, a binary distribution could be installed from a directory containing a setup.cfg, some DLLs, Python source files, and other text and binary data files. Moreover, it would be just as easy to zip that whole directory up (using any zipping tools), and pass it around as a .zip file; at installation time, the packaging code would unpack the directory in a temporary location and install from there. The zip archive can, of course, be appended to an executable which does the relevant unpacking and calls to packaging code to actually do the installation. The scheme is conceptually the same as the wininst-x.y.exe does - only the details differ. This gives a one (double-)click installer. > Agreed - but I'm looking at a pysetup install approach to work for > source and binary packages, essentially replacing the use of > bdist_wininst and bdist_msi with sdist/bdist_simple archives. That's a > change of heart for me, as I used to argue for wininst/msi over > setuptools and similar - but pysetup includes all the listing and > uninstalling features I wanted, so the "one unified approach" has won > me over in preference to the platform integration. Right, but AFAICT pysetup3 will work now with a binary distribution, other than it does not contain mechanisms for checking Python version and platform compatibilities. Being a command line script, it will even support virtual environments without too much trouble - I've been working on this in the pythonv branch with some success. What's missing from things is a .exe installer; even though you might be happy without one, not having it may be seen by some as a retrograde step. > Ideally bdist_wininst and bdist_msi would also integrate with pysetup > and with virtual environments, but I imagine that could be pretty hard > to make work cleanly, as Windows doesn't really support multiple > installations of a software package... I don't think Windows itself cares in general, it's more about the specifics of what's being installed. Obviously some things like COM components would need to be managed centrally, but I would imagine that if you had two projects with separate versions of e.g. C-based extensions, you could install the relevant DLLs in separate virtual environments and not necessarily have problems with them coexisting. > (Plus, I've no real idea about how bdist_wininst works, so while you > may be right, I wouldn't know how to do anything with your suggestion > :-)) Though I can't claim to have looked at the codebase in detail, the overall scheme would appear to be this: bdist_wininst creates an executable from wininst-x.y.exe (part of Python, in a distutils directory), appends some metadata (used in the UI of wininst-x.y.exe - things like the package name, icon etc.) and appends to that an archive containing all the stuff to install. When the executable is run, the UI is presented incorporating relevant metadata, user input solicited and the archive contents installed according to that input. However, the installation locations are determined from the registry information on installed Pythons only, with no nod to the possibility of users having installed multiple virtual environments from those installed Pythons. Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On 10 October 2011 21:38, Vinay Sajip wrote: >> I'm not sure what you mean by a "setup.cfg-based directory". Could > >> you >> clarify, and maybe explain how you'd expect to create such an archive? >> We may be talking at cross-purposes here. > > Here's how I see it: at present, you can install a project by specifying > > pysetup3 install path-to-directory > > where the named directory contains a setup.cfg (replacing setup.py) and a > bunch of things to install. Exactly what to install where is specified in the > setup.cfg: > it covers not only python packages and modules but also arbitrary binary > files. The setup.cfg format is extensible enough to allow specifying where > files are > installed not only declaratively (as static paths in setup.cfg) but also > according to criteria computed at installation time (e.g., write some > PowerShell scripts to > the installing user's PowerShell profile location). > > Of course, since you can install arbitrary data (and record what was > installed where, to allow uninstallation to work), you can certainly install > DLLs too (modulo > the user having write permissions for the installation location, but that's > true for data files, too). > > In theory, therefore, a binary distribution could be installed from a > directory containing a setup.cfg, some DLLs, Python source files, and other > text and binary > data files. Moreover, it would be just as easy to zip that whole directory up > (using any zipping tools), and pass it around as a .zip file; at installation > time, the > packaging code would unpack the directory in a temporary location and install > from there. > > The zip archive can, of course, be appended to an executable which does the > relevant unpacking and calls to packaging code to actually do the > installation. > The scheme is conceptually the same as the wininst-x.y.exe does - only the > details differ. This gives a one (double-)click installer. > Ah, I see what you are saying now. I hadn't realised that the setup.cfg format was that flexible. I'll look into it a bit more - you're right that it would be better to reuse the existing technology than to extend it if that's not needed. >> Agreed - but I'm looking at a pysetup install approach to work for >> source and binary packages, essentially replacing the use of >> bdist_wininst and bdist_msi with sdist/bdist_simple archives. That's a >> change of heart for me, as I used to argue for wininst/msi over >> setuptools and similar - but pysetup includes all the listing and >> uninstalling features I wanted, so the "one unified approach" has won >> me over in preference to the platform integration. > > Right, but AFAICT pysetup3 will work now with a binary distribution, other > than it does not contain mechanisms for checking Python version and platform > compatibilities. > Being a command line script, it will even support virtual environments > without too much trouble - I've been working on this in the pythonv branch > with some success. > What's missing from things is a .exe installer; even though you might be > happy without one, not having it may be seen by some as a retrograde step. > Now I understand what you mean. I agree that an exe installer should be available. And given that it can be used like a zipfile as well if it follows the exe stub plus zipfile approach of bdist_wininst) then that sounds ideal. >> Ideally bdist_wininst and bdist_msi would also integrate with pysetup >> and with virtual environments, but I imagine that could be pretty hard >> to make work cleanly, as Windows doesn't really support multiple >> installations of a software package... > > I don't think Windows itself cares in general, it's more about the specifics > of what's being installed. Obviously some things like COM components would > need to be > managed centrally, but I would imagine that if you had two projects with > separate versions of e.g. C-based extensions, you could install the relevant > DLLs in separate > virtual environments and not necessarily have problems with them coexisting. Agreed, it's more common Windows conventions than windows itself. Plus having to invent distinct names for each entry to go into add/remove programs, which could get to be a pain with multiple venvs. >> (Plus, I've no real idea about how bdist_wininst works, so while you >> may be right, I wouldn't know how to do anything with your suggestion >> :-)) > > Though I can't claim to have looked at the codebase in detail, the overall > scheme would appear to be this: bdist_wininst creates an executable from > wininst-x.y.exe > (part of Python, in a distutils directory), appends some metadata (used in > the UI of wininst-x.y.exe - things like the package name, icon etc.) and > appends to that > an archive containing all the stuff to install. When the executable is run, > the UI is presented incorporating relevant metadata, user input solicited and > the archive > contents installed according to tha
Re: [Python-Dev] Packaging and binary distributions for Python 3.3
On Mon, Oct 10, 2011 at 2:29 PM, Paul Moore wrote: > Ideally bdist_wininst and bdist_msi would also integrate with pysetup > and with virtual environments, but I imagine that could be pretty hard > to make work cleanly, as Windows doesn't really support multiple > installations of a software package... That's OK, the package managers get bypassed by pysetup on POSIX systems as well - that's kind of the point of language level virtual environments (they're an intermediate step between system installs and chroot installs, which in turn are an interim step on the road to full virtualised machines). There are hard to build packages on POSIX (e.g. PIL) that would also benefit from a good, cross-platform approach to binary installation. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Bring new features to older python versions
On 10/10/2011 4:21 PM, Giampaolo Rodolà wrote: Thanks everybody for your feedback. I created a gcode project here: http://code.google.com/p/pycompat/ This project will be easier if the test suite for a particular function/class/module is up to par. If you find any gaping holes, you might file an issue on the tracker. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
