Re: [Python-Dev] Emit a BytesWarning on bytes filenames on Windows
As quoted above, deprecation of the bytes version of the API sounds fine to me, but isn't this going to run into the usual objections from the we need bytes for efficiency crowd? It's OK with mewink to say in this restricted area you must convert to Unicode, but is that going to fly with that constituency? I don't think this we need bytes for efficiency crowd actually exists. We are talking about file names here. The relevant crowd is the we need bytes for correctness, and that crowd focuses primarily on Unix. It splits into the we only care about Unix crowd (A), the we want correctness everywhere crowd (B), and the we want portable code crowd (C). (A) can accept the deprecation. (B) will support it. Only (C) might protest, as we are going to break their code, hence the deprecation period. Regards, Martin ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Emit a BytesWarning on bytes filenames on Windows
On Sun, Oct 30, 2011 at 6:00 PM, Martin v. Löwis mar...@v.loewis.de wrote: As quoted above, deprecation of the bytes version of the API sounds fine to me, but isn't this going to run into the usual objections from the we need bytes for efficiency crowd? It's OK with mewink to say in this restricted area you must convert to Unicode, but is that going to fly with that constituency? I don't think this we need bytes for efficiency crowd actually exists. I think that crowd does exist, but I've only ever seen them complain about URLs and other wire protocols (where turnaround time can matter a lot in terms of responsiveness of network applications for short requests, and encode()/decode() cycles can really add up). Filesystem access is dominated by I/O time, and there's often going to be some encoding or decoding going anyway (since the app and the filesystem have to get the data into a common format). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft PEP: virtual environments
Nick Coghlan ncoghlan at gmail.com writes: All the core committers can actually publish PEPs via the PEP hg repo, so Vinay could probably handle pushing the updates to python.org. Submission via the PEP editors is mainly there as a backstop for cases where there's no current core dev directly involved in the PEP. Added as PEP 404 - hope y'all can find it ;-) Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft PEP: virtual environments
Antoine Pitrou solipsis at pitrou.net writes: Why would that be a problem? Do you plan to install several versions of Python in a single VE? No, but some packages might install headers in /include and others in /include/pythonX.Y. I wasn't sure whether this would cause a problem with files not being found during build, though I realise this can be worked around with specific -I flags to the compiler. At present, we only create a /include in the venv, but not /include/pythonX.Y. We already have Unix shell scripts and BAT files in the source tree. Is it really complicated to maintain these additional shell scripts? Is there a lot of code in them? No, they're pretty small: wc -l gives 76 posix/activate (Bash script, contains deactivate() function) 31 nt/activate.bat 17 nt/deactivate.bat The question is whether we should stop at that, or whether there should be support for tcsh, fish etc. such as virtualenv provides. IMO, if we provide the above as a bare minimum + an easy way for third-party tools to install replacements/additions, then we probably don't need to worry too much about an additional support burden in the stdlib - third parties can take up the responsibility for supporting additional shells or helper scripts. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft PEP: virtual environments
On Sun, 30 Oct 2011 12:10:18 + (UTC) Vinay Sajip vinay_sa...@yahoo.co.uk wrote: We already have Unix shell scripts and BAT files in the source tree. Is it really complicated to maintain these additional shell scripts? Is there a lot of code in them? No, they're pretty small: wc -l gives 76 posix/activate (Bash script, contains deactivate() function) 31 nt/activate.bat 17 nt/deactivate.bat The question is whether we should stop at that, or whether there should be support for tcsh, fish etc. such as virtualenv provides. I don't think we need additional support for more or less obscure shells. Also, if posix/activate is sufficiently well written (don't ask me how :-)), it should presumably be compatible with all Unix shells? Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft PEP: virtual environments
Antoine Pitrou solipsis at pitrou.net writes: We already have Unix shell scripts and BAT files in the source tree. Do we have a blessed location in the stdlib for data files in general? Although we're talking in this instance about scripts, they're just data as far as the venv module is concerned. While it's not uncommon for data which is included with packages to be installed in the source tree for that package(e.g. packaging's test data), I'm not sure what one would do with data which belongs to a top-level module. At the moment it's in the source as a base64-encoded string, but I'm not sure that's ideal - it's workable only because the data is so small. I don't really want to add a Lib/scripts.zip adjacent to venv.py, which venv accesses via os.path.dirname(__file__), because if every module did this, it would be a tad untidy. The other alternative would be to make venv a package with all its code in venv/__init__.py and a scripts.zip adjacent to that. Does that seem like a better solution? Can anyone suggest better alternatives? Sorry if this has come up before and I've missed something obvious. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft PEP: virtual environments
On Sun, 30 Oct 2011 12:35:20 + (UTC) Vinay Sajip vinay_sa...@yahoo.co.uk wrote: The other alternative would be to make venv a package with all its code in venv/__init__.py and a scripts.zip adjacent to that. Does that seem like a better solution? Please don't make it a zip file. We want code to be easily trackable and editable. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Packaging and binary distributions
I'd like to reopen the discussions on how the new packaging module will handle/support binary distributions in Python 3.3. The previous thread (see http://mail.python.org/pipermail/python-dev/2011-October/113956.html) included a lot of good information and discussion, but ultimately didn't reach any firm conclusions. First question - is this a Windows only problem, or do Unix/MacOS users want binary support? My feeling is that it's not an issue for them, at least not enough that anyone has done anything about it in the past, so I'll focus on Windows here. Second question - is there a problem at all? For the majority of Windows users, I suspect not. The existing bdist_wininst and bdist_msi formats have worked fine for a long time, offer Windows integration and a GUI installer, and in the case of MSI offer options for integrating with corporate distribution policies that some users consider significant, if not essential. (Binary eggs are a third, and somewhat odd, case - a number of projects have started distributing binary eggs, but I don't know what benefits they have over bdist_wininst in particular, as easy_install will read bdist_wininst installers. Perhaps a setuptools/distribute user could comment. For now I'll assume that binary eggs will slowly go away as packaging gets more widely adopted). So that leaves a minority who (1) prefer integration with packaging, (2) need to work with virtual environments or custom local builds, (3) need binary extensions in some or all of their environments and (4) don't want to have to build all the binaries they need from scratch. Given the scale of the issue, it seems likely that putting significant effort into addressing it is unwise. In particular, it seems unlikely that developers are going to move en masse to a new distribution format just to cater for this minority. On the other hand, for people who care, the fact that packaging (currently) offers no direct support for consuming binary distributions is a fairly obvious hole. And having to build from source just to install into a virtual environment could be a showstopper. The bdist_wininst format is relatively amenable to manipulation - it's little more than a zip file, after all. So writing 3rd party code to install the contents via packaging shouldn't be hard (I've done some proof of concept work, and it isn't :-)) Vinay's proposal to use the resource mechanism and some custom hooks would work, but I'd like to see a small amount of extra direct support added to packaging to make things cleaner. Also, if packaging supported plugins to recognise new distribution formats, this would make it possible to integrate the extra code seamlessly. The MSI format is a little more tricky, mainly because it is a more complex format and (as far as I can tell from a brief check) files are stored in the opaque CAB format, so the only way of getting data out is to do a temporary install somewhere. But I see no reason why that isn't achievable. So, my proposal is as follows: 1. I will write a 3rd party module to take bsist_wininst and bdist_msi modules and install them using packaging 2. Where packaging changes are useful to make installing binaries easier, I'll request them (by supplying patches) 3. I'll look at creating a format-handling plugin mechanism for packaging. If it's viable, I'll post patches 4. If it seems useful, my module could be integrated into the core packaging module I don't intend to do anything about a GUI, or modify the existing formats at all. These don't interest me, particularly, so I'll leave them to someone who has a clear picture of what they want in those areas, and the time to develop it. For 3.3 at least, I'd expect developers to continue distributing bdist_wininst or bdist_msi format files. We'll see what happens with binary eggs. Unix/MacOS users who care will need to propose something themselves. Does anyone have any comments? Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft PEP: virtual environments
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/30/2011 08:35 AM, Vinay Sajip wrote: Antoine Pitrou solipsis at pitrou.net writes: We already have Unix shell scripts and BAT files in the source tree. Do we have a blessed location in the stdlib for data files in general? Although we're talking in this instance about scripts, they're just data as far as the venv module is concerned. While it's not uncommon for data which is included with packages to be installed in the source tree for that package(e.g. packaging's test data), I'm not sure what one would do with data which belongs to a top-level module. At the moment it's in the source as a base64-encoded string, but I'm not sure that's ideal - it's workable only because the data is so small. I don't really want to add a Lib/scripts.zip adjacent to venv.py, which venv accesses via os.path.dirname(__file__), because if every module did this, it would be a tad untidy. The other alternative would be to make venv a package with all its code in venv/__init__.py and a scripts.zip adjacent to that. Does that seem like a better solution? Can anyone suggest better alternatives? Sorry if this has come up before and I've missed something obvious. +1 to making it a package and keeping the data in the package. - -1 to a zip file: each scripts should be a noraml, version-controlled entity. Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6tb+8ACgkQ+gerLs4ltQ463wCfZoOOYK1c7XgAaihSdM9+0dxn /YgAoMVlq+ZRGA6xZUFNrajSbdr4aUQZ =P6zT -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft PEP: virtual environments
Antoine Pitrou solipsis at pitrou.net writes: Please don't make it a zip file. We want code to be easily trackable and editable. Of course. I was thinking of a directory tree in the source, subject to our normal revision control, but processed during make or installation to be available as a zip file once deployed. It was a general point about data that I was making; in this particular case, that data just happens to be source code. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] cpython (merge 3.2 - default): Fix the return value of set_discard (issue #10519)
On Sun, 30 Oct 2011 13:38:35 +0100 petri.lehtinen python-check...@python.org wrote: http://hg.python.org/cpython/rev/f634102aca01 changeset: 73204:f634102aca01 parent: 73201:a5c4ae15b59d parent: 73203:b643458a0108 user:Petri Lehtinen pe...@digip.org date:Sun Oct 30 14:35:39 2011 +0200 summary: Fix the return value of set_discard (issue #10519) I get the following compiler warning here: Objects/setobject.c: In function ‘set_discard’: Objects/setobject.c:1909:24: attention : unused variable ‘result’ Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft PEP: virtual environments
On Sun, 30 Oct 2011 15:42:11 + (UTC) Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Antoine Pitrou solipsis at pitrou.net writes: Please don't make it a zip file. We want code to be easily trackable and editable. Of course. I was thinking of a directory tree in the source, subject to our normal revision control, but processed during make or installation to be available as a zip file once deployed. It would be even simpler not to process it at all, but install the scripts as-is (without the execute bit) :) Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions
In article cacac1f-cmbkryagzrcawdndm7-vn4yjo99fbd9vvccmbhcv...@mail.gmail.com, Paul Moore p.f.mo...@gmail.com wrote: I'd like to reopen the discussions on how the new packaging module will handle/support binary distributions in Python 3.3. The previous thread (see http://mail.python.org/pipermail/python-dev/2011-October/113956.html) included a lot of good information and discussion, but ultimately didn't reach any firm conclusions. First question - is this a Windows only problem, or do Unix/MacOS users want binary support? My feeling is that it's not an issue for them, at least not enough that anyone has done anything about it in the past, so I'll focus on Windows here. I haven't been following this discussion that closely but I'm rather surprised that the need for binary distributions for Python packages on non-Windows platforms would be in question. Just as on Windows, it's not a given that all Unix or Mac OS X end-user systems will have the necessary development tools installed (C compiler, etc) to build C extension modules. Today, the most platform-independent way of distributing these are with binary eggs: the individual binary eggs are, of course, not platform-independent but the distribution and installation mechanism is or should be. Sure, there are other ways, like pushing the problem back to the OS distributor (e.g. Debian, Red Hat, et al) or, as in the case of Mac OS X where there isn't a system package manager in the same sense, to a third-party package distributor (like MacPorts, Homebrew, or Fink). Or you can produce platform-specific installers for each platform which also seems heavy-weight. Has anyone analyzed the current packages on PyPI to see how many provide binary distributions and in what format? -- Ned Deily, n...@acm.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/30/2011 02:04 PM, Ned Deily wrote: In article cacac1f-cmbkryagzrcawdndm7-vn4yjo99fbd9vvccmbhcv...@mail.gmail.com, Paul Moore p.f.mo...@gmail.com wrote: I'd like to reopen the discussions on how the new packaging module will handle/support binary distributions in Python 3.3. The previous thread (see http://mail.python.org/pipermail/python-dev/2011-October/113956.html) included a lot of good information and discussion, but ultimately didn't reach any firm conclusions. First question - is this a Windows only problem, or do Unix/MacOS users want binary support? My feeling is that it's not an issue for them, at least not enough that anyone has done anything about it in the past, so I'll focus on Windows here. I haven't been following this discussion that closely but I'm rather surprised that the need for binary distributions for Python packages on non-Windows platforms would be in question. Just as on Windows, it's not a given that all Unix or Mac OS X end-user systems will have the necessary development tools installed (C compiler, etc) to build C extension modules. Today, the most platform-independent way of distributing these are with binary eggs: the individual binary eggs are, of course, not platform-independent but the distribution and installation mechanism is or should be. Sure, there are other ways, like pushing the problem back to the OS distributor (e.g. Debian, Red Hat, et al) or, as in the case of Mac OS X where there isn't a system package manager in the same sense, to a third-party package distributor (like MacPorts, Homebrew, or Fink). Or you can produce platform-specific installers for each platform which also seems heavy-weight. Has anyone analyzed the current packages on PyPI to see how many provide binary distributions and in what format? Practically speaking, nobody but Windows consumers *needs* binary packages on PyPI: even if the target (production) box is crippled^Wstripped of its compiler, such environments always have staging hosts which can be used to build binary packages for internal distribution. Windows users are the only ones who routinely don't have access to a compiler at all. Even trying to push binary distributeions to PyPI for Linux is a nightmare (e.g., due to UCS2 / UCS4 incompatibility). Tres. - -- === Tres Seaver +1 540-429-0999 tsea...@palladion.com Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6tvk4ACgkQ+gerLs4ltQ7zLwCfa0tvsRUtkwC3OkhYwGD7eGvL pbwAoLAm416vdyS3qbGDf/2R9iEtw2rH =tcS+ -END PGP SIGNATURE- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Emit a BytesWarning on bytes filenames on Windows
Le 30/10/2011 09:00, Martin v. Löwis a écrit : As quoted above, deprecation of the bytes version of the API sounds fine to me, but isn't this going to run into the usual objections from the we need bytes for efficiency crowd? It's OK with mewink to say in this restricted area you must convert to Unicode, but is that going to fly with that constituency? I don't think this we need bytes for efficiency crowd actually exists. We are talking about file names here. The relevant crowd is the we need bytes for correctness, and that crowd focuses primarily on Unix. Oh, by the way, it is important to know that Unicode filenames is the best way to write portable programs with Python 3. On UNIX, since Python 3.1, undecodables filename don't raise Unicode errors: undecodable bytes are stored as surrogates (see the PEP 383). So even if the computer is completly misconfigured, it just works. On Windows, you must Unicode for filenames for correctness. Anyway, with Python 3, it's easier to manipulate Unicode strings than bytes strings. Martin finally agreed with me, I should hurry to implement my idea! :-) Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Emit a BytesWarning on bytes filenames on Windows
Le 29/10/2011 07:47, Mark Hammond a écrit : When previously discussing this issue, I was under the impression that the problem was unencodable bytes passed from the Python code to Windows - but the reverse is true - only the data coming back from Windows isn't encodable. The undecodable filenames issue occurs mostly on os.listdir(bytes) and os.getcwdb(). Unencodable filenames issue occurs on the rest of my function list. As the data came externally, the only solution the programmer has is to change to the unicode version of the api - so we recommend the bytes version not be used by anyone, anytime - which means it is conceptually deprecated already. I proposed to raise a Unicode error on undecodable filenames, instead of returning invalid filenames (with question marks), to force the developer to move to the Unicode API. But as I explained in my previous message, you have to wait for an user having the problem to be noticied of the problem. Terry J. Reedy is also concerned about backward compatibility (3.2 - 3.3). Emiting a warning, disabled by default, is a softer solution :-) Therefore, as you imply, I think the solution to this issue is to start the process of deprecating the bytes version of the api in py3k with a view to removing it completely - possibly with a less aggressive timeline than normal. If there is a warning, I don't really care of removing the bytes API before Python 4. PendingDeprecationgWarning can be used, or maybe a DeprecationWarning mentioning that the code will stay for long time. In Python 2.7, I think documenting the issue and a recommendation to always use unicode is sufficient (ie, we can't deprecate it and a new BytesWarning seems gratuitous.) Sorry, I don't understand gratuitous here: do you mean that a new warning would annoying, and that it is cheap and useful to add it to Python 2.7.x? Victor ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft PEP: virtual environments
Antoine Pitrou solipsis at pitrou.net writes: It would be even simpler not to process it at all, but install the scripts as-is (without the execute bit) :) Sure, but such an approach makes it difficult to provide a mechanism which is easily extensible; for example, with the current approach, it is straightforward for third party tools to either easily replace completely, update selectively or augment simply the scripts provided by base classes. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions
On 30 October 2011 18:04, Ned Deily n...@acm.org wrote: Has anyone analyzed the current packages on PyPI to see how many provide binary distributions and in what format? A very quick and dirty check: dmg: 5 rpm: 12 msi: 23 dumb: 132 wininst: 364 egg: 2570 That's number of packages with binary distributions in that format. It's hard to be sure about egg distributions, as many of these could be pure-python (there's no way I know, from the PyPI metadata, to check this). This is 2913 packages with some form of binary distribution out of 16615 that have a single release on PyPI. I skipped 398 with multiple releases as I wasn't sure how to capture the data for those... I suspect they include some important cases, though (I know lxml is in there, for example). So: 17% of packages have any binary release. Of those, 88% have eggs, 12% have wininst and the rest are under 5%. Put another way, 2% of all packages have wininst installers. And 15% have eggs. That's not a lot. Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] draft PEP: virtual environments
On Sun, 30 Oct 2011 22:47:13 + (UTC) Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Antoine Pitrou solipsis at pitrou.net writes: It would be even simpler not to process it at all, but install the scripts as-is (without the execute bit) :) Sure, but such an approach makes it difficult to provide a mechanism which is easily extensible; for example, with the current approach, it is straightforward for third party tools to either easily replace completely, update selectively or augment simply the scripts provided by base classes. I don't understand why a zip file makes this easier (especially the update selectively part). Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions
Paul Moore p.f.moore at gmail.com writes: The MSI format is a little more tricky, mainly because it is a more complex format and (as far as I can tell from a brief check) files are stored in the opaque CAB format, so the only way of getting data out is to do a temporary install somewhere. But I see no reason why that isn't achievable. It's not just about getting the data out of the CAB, though - it's also about integration with Add/Remove Programs and the rest of the Windows Installer ecosystem. 1. I will write a 3rd party module to take bsist_wininst and bdist_msi modules and install them using packaging It would be important to retain the flexibility currently offered by setup.cfg hooks, as I don't believe any out-of-the-box approach will work for the wide range of use cases on Windows (think Powershell scripts, Visio templates and other Microsoft Office integration components). I'm also not sure if these formats provide all the flexibility required - e.g. they may be fine for extension modules, but how do they handle packaging include files? For 3.3 at least, I'd expect developers to continue distributing bdist_wininst or bdist_msi format files. We'll see what happens with binary eggs. Unix/MacOS users who care will need to propose something themselves. I'm not sure there's anything especially Windows-specific about the bdist_wininst format, apart from the prepended GUI executable. One drawback of any current scheme is that if you're packaging an extension module that runs on say Windows, Linux and Mac OS X, there's no easy way to build or distribute a single archive (for a given version of Python, say) which has all the binary variants you want to include, such that at installation time, only the bits relevant to the target platform are installed. The current packaging functionality does sort of support this, but it entails potentially tedious manual editing of the setup.cfg file to add information about what resources apply to which platform - the kind of tedious editing which would be obviated by the right kind of additional support code. Regards, Vinay Sajip ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Packaging and binary distributions
I like binary distribution even under Linux. I access some Linux machines using same Linux distribution and some of them doesn't have python-dev package or even build-essensials. (because they are netbooting so have restricted rootfs size) So I want build binary package by myself and distribute it to virtualenv on such machines. In this case, absolute path of virtualenv is not fixed. So bdist_dumb --relative or egg is good for me. On Sun, Oct 30, 2011 at 11:09 PM, Paul Moore p.f.mo...@gmail.com wrote: I'd like to reopen the discussions on how the new packaging module will handle/support binary distributions in Python 3.3. The previous thread (see http://mail.python.org/pipermail/python-dev/2011-October/113956.html) included a lot of good information and discussion, but ultimately didn't reach any firm conclusions. First question - is this a Windows only problem, or do Unix/MacOS users want binary support? My feeling is that it's not an issue for them, at least not enough that anyone has done anything about it in the past, so I'll focus on Windows here. Second question - is there a problem at all? For the majority of Windows users, I suspect not. The existing bdist_wininst and bdist_msi formats have worked fine for a long time, offer Windows integration and a GUI installer, and in the case of MSI offer options for integrating with corporate distribution policies that some users consider significant, if not essential. (Binary eggs are a third, and somewhat odd, case - a number of projects have started distributing binary eggs, but I don't know what benefits they have over bdist_wininst in particular, as easy_install will read bdist_wininst installers. Perhaps a setuptools/distribute user could comment. For now I'll assume that binary eggs will slowly go away as packaging gets more widely adopted). So that leaves a minority who (1) prefer integration with packaging, (2) need to work with virtual environments or custom local builds, (3) need binary extensions in some or all of their environments and (4) don't want to have to build all the binaries they need from scratch. Given the scale of the issue, it seems likely that putting significant effort into addressing it is unwise. In particular, it seems unlikely that developers are going to move en masse to a new distribution format just to cater for this minority. On the other hand, for people who care, the fact that packaging (currently) offers no direct support for consuming binary distributions is a fairly obvious hole. And having to build from source just to install into a virtual environment could be a showstopper. The bdist_wininst format is relatively amenable to manipulation - it's little more than a zip file, after all. So writing 3rd party code to install the contents via packaging shouldn't be hard (I've done some proof of concept work, and it isn't :-)) Vinay's proposal to use the resource mechanism and some custom hooks would work, but I'd like to see a small amount of extra direct support added to packaging to make things cleaner. Also, if packaging supported plugins to recognise new distribution formats, this would make it possible to integrate the extra code seamlessly. The MSI format is a little more tricky, mainly because it is a more complex format and (as far as I can tell from a brief check) files are stored in the opaque CAB format, so the only way of getting data out is to do a temporary install somewhere. But I see no reason why that isn't achievable. So, my proposal is as follows: 1. I will write a 3rd party module to take bsist_wininst and bdist_msi modules and install them using packaging 2. Where packaging changes are useful to make installing binaries easier, I'll request them (by supplying patches) 3. I'll look at creating a format-handling plugin mechanism for packaging. If it's viable, I'll post patches 4. If it seems useful, my module could be integrated into the core packaging module I don't intend to do anything about a GUI, or modify the existing formats at all. These don't interest me, particularly, so I'll leave them to someone who has a clear picture of what they want in those areas, and the time to develop it. For 3.3 at least, I'd expect developers to continue distributing bdist_wininst or bdist_msi format files. We'll see what happens with binary eggs. Unix/MacOS users who care will need to propose something themselves. Does anyone have any comments? Paul. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com -- INADA Naoki songofaca...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
Re: [Python-Dev] Packaging and binary distributions
On 30 October 2011 23:17, Vinay Sajip vinay_sa...@yahoo.co.uk wrote: Paul Moore p.f.moore at gmail.com writes: The MSI format is a little more tricky, mainly because it is a more complex format and (as far as I can tell from a brief check) files are stored in the opaque CAB format, so the only way of getting data out is to do a temporary install somewhere. But I see no reason why that isn't achievable. It's not just about getting the data out of the CAB, though - it's also about integration with Add/Remove Programs and the rest of the Windows Installer ecosystem. Hang on. I'm talking here about repackaging the binary files in the MSI file for use in a pysetup install invocation. As pysetup has no GUI, and doesn't integrate with Add/Remove, there's no issue here. If you want a GUI and Add/Remove integration, just run the MSI. Or am I missing something? We seem to be at cross purposes here, I suspect I'm missing your point. 1. I will write a 3rd party module to take bsist_wininst and bdist_msi modules and install them using packaging It would be important to retain the flexibility currently offered by setup.cfg hooks, as I don't believe any out-of-the-box approach will work for the wide range of use cases on Windows (think Powershell scripts, Visio templates and other Microsoft Office integration components). Why? Again, if this is purely as a means to consume bdist_xxx files, then the only flexibility needed is enough to cater for any variations in data stored in the bdist_xxx format. The wininst format is easy here - it has directories PLATLIB, PURELIB, DATA, SCRIPTS and HEADERS (corresponding to the installation --install-xxx parameters) and that's all. As long as the module is flexible enough to deal with that, it can read anything bdist_wininst can produce. I'm also not sure if these formats provide all the flexibility required - e.g. they may be fine for extension modules, but how do they handle packaging include files? Ah, I think I see what you are getting at. If someone uses the new features and flexibility of packaging to create a fancy custom install scheme, how do they bundle up a binary distribution from that? My (current) answer is that I don't know. The packaging module as it stands only offers the legacy bdist_xxx formats, so the answer is run pysetup run bdist_wininst on it. If that breaks (as it is likely to - wininst format isn't very flexible) then tough, you're out of luck. I 100% agree that having a native packaging means of building binary distributions from source ones, which captures all of the necessary information to cover any flexibility available to setup.cfg, would be good. But that's potentially a much bigger project than I can manage. My bdist_simple format was based off bdist_dumb/bdist_wininst and had the same limitations as that. You might be able to get somewhere by running build, then zipping up the whole directory, source, build subdirectory and all. Then on the target machine, unzip and do a --skip-build install. That's a bit of a hack, but should in theory work. Whether it's the basis of a sensible distribution format I don't know. For 3.3 at least, I'd expect developers to continue distributing bdist_wininst or bdist_msi format files. We'll see what happens with binary eggs. Unix/MacOS users who care will need to propose something themselves. I'm not sure there's anything especially Windows-specific about the bdist_wininst format, apart from the prepended GUI executable. One drawback of any current scheme is that if you're packaging an extension module that runs on say Windows, Linux and Mac OS X, there's no easy way to build or distribute a single archive (for a given version of Python, say) which has all the binary variants you want to include, such that at installation time, only the bits relevant to the target platform are installed. The current packaging functionality does sort of support this, but it entails potentially tedious manual editing of the setup.cfg file to add information about what resources apply to which platform - the kind of tedious editing which would be obviated by the right kind of additional support code. Again, I agree that this would be useful. Not something I have the time to look at though (although if someone else picks it up, I'd be interested in doing some testing and maybe contributing to the work). I think I now see why we're not understanding each other. I'm coming from the position that the projects I care about (as an end user) use bdist_wininst or bdist_msi at the moment, so all I want is a way of using, as a consumer, those existing distributions (or something equivalent in power) to install the packages via pysetup (which gets me the ability to install in development builds and venvs). I see why a more powerful binary format would be nice for developers, but as an end user I have no direct need for it. Thanks for your patience. Paul.
Re: [Python-Dev] Emit a BytesWarning on bytes filenames on Windows
On 31/10/2011 8:39 AM, Victor Stinner wrote: Le 29/10/2011 07:47, Mark Hammond a écrit : When previously discussing this issue, I was under the impression that the problem was unencodable bytes passed from the Python code to Windows - but the reverse is true - only the data coming back from Windows isn't encodable. The undecodable filenames issue occurs mostly on os.listdir(bytes) and os.getcwdb(). Unencodable filenames issue occurs on the rest of my function list. As the data came externally, the only solution the programmer has is to change to the unicode version of the api - so we recommend the bytes version not be used by anyone, anytime - which means it is conceptually deprecated already. I proposed to raise a Unicode error on undecodable filenames, instead of returning invalid filenames (with question marks), to force the developer to move to the Unicode API. But as I explained in my previous message, you have to wait for an user having the problem to be noticied of the problem. Terry J. Reedy is also concerned about backward compatibility (3.2 - 3.3). Emiting a warning, disabled by default, is a softer solution :-) Right - and just to be clear, we are all now agreeing that the UnicodeDecodeError isn't appropriate and a warning will be issued instead? Therefore, as you imply, I think the solution to this issue is to start the process of deprecating the bytes version of the api in py3k with a view to removing it completely - possibly with a less aggressive timeline than normal. If there is a warning, I don't really care of removing the bytes API before Python 4. Agreed - I was trying to say that I think we should start the deprecation process of the bytes API, so a [Pending]DeprecationWarning would then be appropriate. The actual timing of the removal isn't important. PendingDeprecationgWarning can be used, or maybe a DeprecationWarning mentioning that the code will stay for long time. In Python 2.7, I think documenting the issue and a recommendation to always use unicode is sufficient (ie, we can't deprecate it and a new BytesWarning seems gratuitous.) Sorry, I don't understand gratuitous here: do you mean that a new warning would annoying, and that it is cheap and useful to add it to Python 2.7.x? I mean Uncalled for; lacking good reason; unwarranted. IOW, I don't think we need to take any action for 2.7, apart from possibly documentation changes. Mark ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Emit a BytesWarning on bytes filenames on Windows
On 10/30/2011 5:39 PM, Victor Stinner wrote: Terry J. Reedy is also concerned about backward compatibility (3.2 - 3.3). Emiting a warning, disabled by default, is a softer solution :-) The fact that Mark, Martin, and someone else, I believe, agree with you that the bytes api is not useful at all in 3.x and should go away reduces my concern. This fact does suggest that it is not worth changing anything to make those APIs easier to use. Instead, better to encourage people to not use those APIs in any 3.x code. Removal is ultimately, of course, the hardest solution. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
