Re: [Python-Dev] Python install layout and the PATH on win32
Mark, MAL, Martin, Tarek, Could you comment on this? This is in the context of changing the name of the 'Scripts' directory on windows to 'bin'. Éric brings up the point (explained more below) that if we make this change, packages made/installed the new packaging infrastructure and those made/installed with bdist_winist and the old (frozen) distutils will be inconsistent. The reason why is that the old distutils has a hard-coded dict in distutils.command.install that would point to the old locations. If we were to make this change in sysconfig.cfg, we would probably want to make a corresponding change in the INSTALL_SCHEMES dict in distutils.command.install. More context: On 3/20/2012 10:41 PM, Éric Araujo wrote: Le 20/03/2012 21:40, VanL a écrit : On Tuesday, March 20, 2012 at 5:07 PM, Paul Moore wrote: It's worth remembering Éric's point - distutils is frozen and changes are in theory not allowed. This part of the proposal is not possible without an exception to that ruling. Personally, I don't see how making this change could be a problem, but I'm definitely not an expert. If distutils doesn't change, bdist_wininst installers built using distutils rather than packaging will do the wrong thing with regard to this change. End users won't be able to tell how an installer has been built. Looking at the code in bdist_wininst, it loops over the keys in the INSTALL_SCHEMES dict to find the correct locations. If the hard-coded dict were changed, then the installer would 'just work' with the right location - and this matches my experience having made this sort of change. When I change the INSTALL_SCHEMES dict, things get installed according to the new scheme without difficulty using the standard tools. The only time when something is trouble is if it does its own install routine and hard-codes 'Scripts' as the name of the install directory - and I have only seen that in PyPM a couple versions ago. From the top of my head the developers with the most experience about Windows deployment are Martin v. Löwis, Mark Hammond and Marc-André Lemburg (not sure about the Windows part for MAL, but he maintains a library that extends distutils and has been broken in the past). I think their approval is required for this kind of huge change. Note the above - this is why I would like your comment. The point of the distutils freeze (i.e. feature moratorium) is that we just can’t know what complicated things people are doing with undocumented internals, because distutils appeared unmaintained and under-documented for years and people had to work with and around it; since the start of the distutils2 project we can Just Say No™ to improvements and features in distutils. “I don’t see what could possibly go wrong” is a classic line in both horror movies and distutils developmentwink. Renaming Scripts to bin on Windows would have effects on some tools we know and surely on many tools we don’t know. We don’t want to see again people who use or extend distutils come with torches and pitchforks because internals were changed and we have to revert. So in my opinion, to decide to go ahead with the change we need strong +1s from the developers I named above and an endorsement by Tarek, or if he can’t participate in the discussion, Guido. As a footnote, distutils is already broken in 3.3. Now we give users or system administrators the possibility to edit the install schemes at will in sysconfig.cfg, but distutils hard-codes the old scheme. I tend to think it should be fixed, to make the distutils-packaging transition/cohabitation possible. Any comment? Thanks, Van CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 5:48 AM, Mark Hammond wrote: While I'm still unclear on the actual benefits of this, Martin's approach strikes a reasonable compromise so I withdraw my objections. Ok. I was out of town and so could not respond to most of the latest discussion. A question for you Mark, Paul, (and anyone else): Éric correctly points out that there are actually two distinct changes proposed here: 1. Moving the Python binary 2. Changing from Scripts to bin So far, the primary resistance seems to be to item #1 - moving the python binary. There have been a few people who have noted that #2 will require some code to change (i.e. Paul), but I don't see lots of resistance. Am I reading you correctly? Thanks, VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 1:19 PM, Paul Moore wrote:
Somewhat. I don't really object to #1, but mildly object to #2. I also
note that the proposals round the Lib directory seem to have
disappeared. I assume those have been dropped - they were the ones I
did object to.
They are of secondary importance to me, and I would be mostly ok to drop
them - but I would like to understand your objection.
I would like to know if you would object to user lib installs matching
the system install. I.e., would it cause problems with you if it were
just 'lib' everywhere, with no 'lib/python{version}'? It sounded like
adding the version directory was the issue.
Thanks,
VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by
U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any
U.S. tax advice contained in this communication (including any
attachments) was not intended or written to be used, and cannot be
used, for the purpose of (i) avoiding penalties under the Internal
Revenue Code or (ii) promoting, marketing or recommending to another
party any transaction or matter addressed herein.
CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential,
may be privileged and should be read or retained only by the intended
recipient. If you have received this transmission in error, please
immediately notify the sender and delete it from your system.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/20/2012 3:31 PM, Paul Moore wrote: Serious question: Given a brand new PC, if you were installing Python 2.7, 3.2, 3.3a1, jython, and pypy, what would you do (beyond simply running 5 installers) to get your environment set up the way you want? I install each python in its own directory: C:/lib/python/2.7 C:/lib/python/3.2 C:/lib/python/3.3 C:/lib/jython C:/lib/pypy Jython and Pypy get their own directories because they can have different version compatibilities. I then edit my distutils.command.install and patch pip/virtualenv so that all my directories are 'bin'/'lib'/'include'. I have never used the py.exe runner, but I then choose whichever Python is my default (right now 2.7, but hoping that I will be able to switch during the 3.3 timeframe) and that gets put on the PATH, along with its 'bin' directory. The other root dirs/bin directories get put on the PATH after the default Python. I don't remember whether I did it or whether it is installed that way, but I have a python2.6.exe and an pythonw.2.6.exe, etc, and all the individual installers include both a pip and a pip-2.6 version (or whatever that install has). I honestly don't remember - I would love to have someone else check. With this setup, I get my default choice anytime I type python and a specific interpreter version when I specify it. Same with installers, etc. I then install virtualenv and virtualenvwrapper-powershell and do all of my development out of virtualenvs. Occasionally I will install something to the system python if it is a pain to compile and I am installing a binary version from somewhere, but I generally try to keep the system python(s) clean.CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
Carl - Changing the directory name is in fact a new and different (and much more invasive) special case, because distutils et al install scripts there, and that directory name is part of the distutils install scheme. Installers don't care where the Python binary is located, so moving it in with the other scripts has very little impact. So would changing the distutils install scheme in 3.3 - as defined and declared by distutils - lead to a change in your code? Alternatively stated, do you independently figure out that your virtualenv is on Windows and then put things in Scripts, etc, or do you use sysconfig? If sysconfig gave you different (consistent) values across platforms, how would that affect your code? Thanks, VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/16/2012 3:38 AM, Paul Moore wrote: On 16 March 2012 00:12, Carl Meyerc...@oddbird.net wrote: Changing the directory name is in fact a new and different (and much more invasive) special case, because distutils et al install scripts there, and that directory name is part of the distutils install scheme. Installers don't care where the Python binary is located, so moving it in with the other scripts has very little impact. This is very interesting, as it seems to argue against Mark's point. If moving the Python binary is not an issue here, then would this change make it any more/less of an issue? 1. The incompatibilities between platforms is precisely the problem that sysconfig is designed to solve, isn't it? So tools in Python will either use sysconfig (and be correct regardless of layout) or should be encouraged to change to use sysconfig (so they are layout-independent). Right. I want to change the default layout in sysconfig.cfg. 2. The differences in layout between a installed Python, uninstalled builds and virtualenvs, on the same platform, are more annoying in practice than any cross-platform differences (at least for me). But again, these are known issues that can be dealt with easily enough (trivially via sysconfig from within Python). These differences are a major pain for me - and it doesn't make sense that they should need to be worked around each and every time. If I were tidying up, I would consider renaming Scripts to bin on Windows, and putting the Python executables in there (so there's only one directory to add to PATH, and it uses the common name bin rather than a name that implies that it doesn't contain exes). But that offers no practical benefit... This is not a we should be consistent argument - I know that would never fly. I do cross-platform dev all the time (develop on Windows and Mac, deploy on Linux) and so this bites me *every single time* I want to get a consistent layout between these three. That could be because I want my deployment environment to match my development environment(s), it could be because I need to introspect the layout to find some data, or because I want to check in an entire environment into source control. This is not purely aesthetics - this is an issue I deal with all the time. Thanks, VanCIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/16/2012 10:53 AM, Paul Moore wrote: The only way I can read this to make sense is that you somehow consider the Python installation as part of your development environment (you mentioned source control earlier in the thread - surely you don't manage your Python installation in source control - binaries, stdlib, etc?). I can't see why you would do this, and it certainly doesn't seem like a reasonable thing to do to me. Can you clarify? I don't check in the python binary itself, nor the stdlib, but I *do* check in the whole installation, including the binaries directory. I like having my deploy environment exactly match my develop environment. So if I do have an executable program, I put it in the binaries directory and check it in. My .hgignore includes python, python.exe, pip, easy_install, etc. - things that are owned by the installation - but it includes everything else. As for the stdlib, I don't check that in, so that portion of the proposal (standardize lib naming) is nice to have, but not essential to me. For example, in the following environment: env/ bin/ python pip easy_install my_script lib/ [stuff] data/ [stuff] src/ my_package I would include bin/my_script, src/, and data/ in my version control. This breaks cross-platform development if bin is named Scripts. CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Python install layout and the PATH on win32
On 3/16/2012 11:57 AM, Glenn Linderman wrote: So I think I'm finally beginning to see the underlying reason why Van is desiring this consistency: It is not that he wants to check in his installation of Python, but that he wants to check in his installation of his packages and scripts into a source control environment, and then be able to check out that source control environment into an installation of Python on another machine of a different architecture. In an environment where a source control system is pervasive and well used, this would be an effective deployment alternative to developing a packaging/distribution solution using distutils, distutels2, packaging, easy_install, eggs, or peanuts, or any other such scheme. But! Source control environments don't lend themselves to being used for anything except exact replication of file and directory structure, so when the different architectures have different directory structures, this deployment technique cannot easily work except, as Van has discussed, by tweaking the development machine's environment to match that of the deployment machines... and that only works in the case where the deployment happens to only one architecture, and the development machine can be tweaked to match... but deploying to multiple machine having different architectures and directory structures would be impossible using the source control deployment technique, because of the different directory structures. This is exactly correct.CIRCULAR 230 NOTICE: To ensure compliance with requirements imposed by U.S. Treasury Regulations, Haynes and Boone, LLP informs you that any U.S. tax advice contained in this communication (including any attachments) was not intended or written to be used, and cannot be used, for the purpose of (i) avoiding penalties under the Internal Revenue Code or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein. CONFIDENTIALITY NOTICE: This electronic mail transmission is confidential, may be privileged and should be read or retained only by the intended recipient. If you have received this transmission in error, please immediately notify the sender and delete it from your system. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
