[Python-Dev] Maintaining civility - the core of the Python community

[VanL]

Hello everyone,

You all have probably noted that there have been some contentious threads
recently, ultimately ending in a few people being given a time-out from
posting on these lists.

I don't normally get into things on this list, but it has been generally
discouraging to see a bunch of generally nice and reasonable people get
sidetracked into unproductive, sarcastic, and unhelpful remarks. There are
some efforts underway to formalize what is in and out of bounds - but I
would suggest that we are losing our way when we need to get to explicit
lists of things not to do.

Unfortunately, we are getting there.

I would like to reemphasize that we are bound together by the things that
we share. We love working on Python. We love being in the community and
seeing it grow. We can make our community stronger and more pleasant by
choosing to emphasize the things that we have in common and by ignoring or
avoiding topics that are more likely to generate unproductive discussion.

We can and should also try to remember that not everyone is coming from the
same place, and so we should actively assume the best of others and
interpret what they say in the most charitable way possible. Think of it as
Postel's law [1] as applied to people.

I'd also suggest that generally, 1) use of profanity, 2) use of sexual
terms and imagery, and 3) use of specifically denigrating terms to refer to
a person [2][3][4] are completely out of bounds for the professional
environment we want to maintain. It is ok to attack arguments, and ideas,
but ad hominem arguments - those attack a person, rather than the person's
argument - are also inappropriate. Use of sarcasm should be strongly
moderated, as it is not correctly interpreted by many people.

No reply is needed to this email. Instead, I'd prefer to see a continuation
of solid technical discussion, rather than meta-discussion.

Thanks,
Van


[1] https://en.wikipedia.org/wiki/Robustness_principle
[2] https://en.wikipedia.org/wiki/List_of_ethnic_slurs
[3] https://en.wikipedia.org/wiki/List_of_religious_slurs
[4]
https://en.wikipedia.org/wiki/List_of_disability-related_terms_with_negative_connotations
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] wininst-*.exe files in Lib/distutils/command

[VanL]

Hi all,

I am looking into an issue associated with the wininst-*.exe files in the
distutils/command subdirectory. It looks like these are the executable
stubs used to create self-extracting zips for installation - but I am not
100% sure. It also looks like they include the calls to standard Windows
functions to display the installer window.

I have a couple questions I need help with:
1) Am I correct about the function, and if not, what are they?
2) Where did these come from, and where is their source code?

Thanks,
Van
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] wininst-*.exe files in Lib/distutils/command

[VanL]

Thank you all, this gives me what I need. Sorry I missed the source in the
the PC/ directory.

Thanks,
Van

On Thu, Oct 18, 2018 at 9:41 AM Zachary Ware 
wrote:

> On Thu, Oct 18, 2018 at 9:09 AM VanL  wrote:
> > Hi all,
> >
> > I am looking into an issue associated with the wininst-*.exe files in
> the distutils/command subdirectory. It looks like these are the executable
> stubs used to create self-extracting zips for installation - but I am not
> 100% sure. It also looks like they include the calls to standard Windows
> functions to display the installer window.
> >
> > I have a couple questions I need help with:
> > 1) Am I correct about the function, and if not, what are they?
>
> You are correct.  IIUC, they are checked in to allow creating those
> installers from non-Windows platforms.
>
> > 2) Where did these come from, and where is their source code?
>
> Source can be found here:
> https://github.com/python/cpython/tree/master/PC/bdist_wininst
>
> The individual checked-in .exe files were each originally built by
> whoever updated the Windows toolchain to a new version of MSVC
> (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest
> ones were added by Thomas Heller, presumably using whatever the
> current toolchain(s) was (were) at the time).  A few of them have been
> rebuilt after bug fixes in the source since they were added, mostly by
> the same people, though I also see Mark Hammond and Raymond Hettinger
> in the history (and Georg Brandl via svnmerge).  I notice that there
> are a few very minor code cleanups (the three latest commits here
> https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c
> )
> that have not made it into a rebuilt exe yet.
>
> FTR, we really ought to remove all but the 14.0 version from the
> master branch.  We don't support building Python with any toolchain
> older than 14.0 anymore, and the older toolchains are nigh impossible
> to find anymore anyway.
>
> --
> Zach
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] wininst-*.exe files in Lib/distutils/command

[VanL]

Primarily for non-windows platforms, but I also think for Windows users
without any compilers or similar tools installed. There is also some
discussion of removing some of the older toolchain-specific versions
(leaving only -14), but that is a subject for another day.

Also I am not sure that bug report applies to 3.5/3.6/3.7.

Thanks,
Van


On Thu, Oct 18, 2018 at 11:26 AM Petr Viktorin  wrote:

> On 10/18/18 4:40 PM, Zachary Ware wrote:
> > On Thu, Oct 18, 2018 at 9:09 AM VanL  wrote:
> >> Hi all,
> >>
> >> I am looking into an issue associated with the wininst-*.exe files in
> the distutils/command subdirectory. It looks like these are the executable
> stubs used to create self-extracting zips for installation - but I am not
> 100% sure. It also looks like they include the calls to standard Windows
> functions to display the installer window.
> >>
> >> I have a couple questions I need help with:
> >> 1) Am I correct about the function, and if not, what are they?
> >
> > You are correct.  IIUC, they are checked in to allow creating those
> > installers from non-Windows platforms.
>
> Is that the only reason for them?
> At least on Linux, bdist_wininst does not work since at least Python
> 3.2, as it tries to use a Windows-only encoding internally.
> https://bugs.python.org/issue10945
>
> If they're only there for non-Windows platforms, they're useless.
>
> >> 2) Where did these come from, and where is their source code?
> >
> > Source can be found here:
> > https://github.com/python/cpython/tree/master/PC/bdist_wininst
> >
> > The individual checked-in .exe files were each originally built by
> > whoever updated the Windows toolchain to a new version of MSVC
> > (Christian Heimes, Brian Curtin, or Steve Dower; though the oldest
> > ones were added by Thomas Heller, presumably using whatever the
> > current toolchain(s) was (were) at the time).  A few of them have been
> > rebuilt after bug fixes in the source since they were added, mostly by
> > the same people, though I also see Mark Hammond and Raymond Hettinger
> > in the history (and Georg Brandl via svnmerge).  I notice that there
> > are a few very minor code cleanups (the three latest commits here
> >
> https://github.com/python/cpython/commits/master/PC/bdist_wininst/install.c
> )
> > that have not made it into a rebuilt exe yet.
> >
> > FTR, we really ought to remove all but the 14.0 version from the
> > master branch.  We don't support building Python with any toolchain
> > older than 14.0 anymore, and the older toolchains are nigh impossible
> > to find anymore anyway.
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Roundup improvements use-cases

[VanL]

Mario wrote:

USE CASE F:

Van Lindberg is concerned about code submissions from non-core
developers: how can the PSF re-license this code in the future without
talking to each contributor, whether the PSF is safe from litigation
based on copyrights of these contributions and related questions are
always bugging him. While the PSF has Contributor Agreements exactly
for these cases, it would be great to have the issue tracker and the
VCS talk to each other, so they could ask contributors to sign (or
declare to have already signed) the CAs when necessary.


+1. This is exactly right.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Volunteer needed to organize summits

[VanL]

Have you had any bites?

Thanks,

Van

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PyCon Early Bird is Ending Soon!

[VanL]

Do you have a year-end hole in your training budget? Or will the
improved economy let you finally attend a work conference? Come to sunny
and warm Atlanta in February for PyCon 2010. Early bird registration
ends on January 6.

Register: https://us.pycon.org/2010/register/

See the talks: http://us.pycon.org/2010/conference/talks/
Get trained at a tutorial:  http://us.pycon.org/2010/tutorials/

Also see the five (or more!) talks that people can't miss at PyCon:

PyOraGeek: PyCon pre-favorites


Pyright: PyCon pre-favorites, the Carl T. edition:


Aftermarket Pipes: Five Pycon 2010 Talks I Need to See:


Jessenoller.com: PyCon 2010: Talks I want to see:


The Third Bit: Five PyCon Talks I Want To See:


See you at PyCon!

Register: https://us.pycon.org/2010/register/

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PyCon is coming! Tomorrow, Feb. 10th is the last day for pre-conference rates

[VanL]

PyCon is coming! Tomorrow (February 10th) is the last day for
pre-conference rates. You can register for PyCon online at:



Register while it is still Feb. 10th somewhere in the world and rest
easy in the knowledge that within 10 days you will enjoying the company
of some of the finest Python hackers in the world.

As an additional bonus, PyCon this year will be in Atlanta, making it an
ideal location for those looking for a way to escape the late winter
blizzards in the northeastern United States, or the dreary fog of the
Bay area.

See you at PyCon 2010!

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Possible patch for functools partial - Interested?

[VanL]

Howdy all -

I have an app where I am using functools.partial to bundle up jobs to
do, where a job is defined as a callable + args. In one case, I wanted
to keep track of whether I had previously seen a job, so I started
putting them into a set... only to find out that partials never test
equal to each other:

>>> import operator
>>> from functools import partial
>>> p1 = partial(operator.add)
>>> p2 = partial(operator.add)
>>> p1 == p2
False
>>> seen = set();seen.add(p1)
>>> p2 in seen
False

I created a subclass of functools.partial that provides appropriate
__eq__ and __hash__ methods, so that this works as expected. I called
the subclass a Job:
>>> j1 = Job(operator.add)
>>> j2 = Job(operator.add)
>>> j1 == j2
True
>>> seen = set();seen.add(j1)
>>> j2 in seen
True
>>> j1 is j2
False

While I was at it, I also added a nice repr. Would this group be
interested in a patch, or is this not interesting?

Thanks,

Van

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Possible patch for functools partial - Interested?

[VanL]

On 5/7/2010 12:41 PM, Steven D'Aprano wrote:

>> Now, if I write
>>
>> def f1(x,y): return x+y
>> def f2(x,y): return x+y
>>
>> I don't expect  f1==f2 to be True, even though f1 and f2 behave in
>> exactly the same way, and indeed it is not.

This is not what I am getting after; these (IMO) should compare unequal.
I took a fairly conservative line, testing for identity of functions and
equality of arguments when converted to a normal form:

def __eq__(self, other):
try:
return ((self.func == other.func) and
(self.args == other.args) and
(self.keywords == other.keywords))
except AttributeError:
return False

def __hash__(self):
if self.keywords:
sorted_keys = sorted(self.keywords.keys())
values = [self.keywords[k] for k in sorted_keys]
kwhash = tuple(sorted_keys + values)
else:
kwhash = None
return hash(tuple((self.func, self.args, kwhash)))

It may be slightly controversial that I read out all the keys and values
and convert them to a normal form for the hash. This is both to get
around hashing issues (keywords is a dict, even though it is read-only)
and to allow for a dict-like object to be provided as an argument (which
happens in my application sometimes).

The repr is slightly long, too, but it was designed to support (weakly)
the idea that obj == exec(repr(obj)). Its not necessary, but I always
like it when that is true.

def __repr__(self):
if self.args: argstr = '*%s' % repr(tuple(self.args))
else: argstr = ''
if self.keywords: kwstr = '**%s' % self.keywords
else: kwstr = ''
if argstr and kwstr: argstr += ', '
classname = self.__class__.__name__
funcname = self.func.__name__
if argstr or kwstr: funcname += ', '
return '%s(%s%s%s)' % (classname, funcname, argstr, kwstr)

I make no effort to look for functional equality between two different
functions. too many side effects could go off trying to make that work,
and it would still be only inconsistently right. (IIRC, doing it
completely is NP-hard.)

Thanks,

Van

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Possible patch for functools partial - Interested?

[VanL]

On 5/11/2010 11:54 PM, Cameron Simpson wrote:
> I know for functions "==" and "is" currently are equivalent, but we should be
> really finicky here about intent, especially since a few messages in the
> thread is contemplate testing function for equivalence to one degree or
> other. At which point "==" and "is" aren't the same any more.
>   

As I stated above, I make no effort to address equivalence of the
functions, nor of the arguments. I am addressing identity of the partial
object, which I am defining as identity of the functions + identity of
the args + equivalence of the keyword arguments after a deterministic
process has been applied.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Licensing // PSF // Motion of non-confidence

[VanL]

On 7/5/2010 11:47 PM, Antoine Pitrou wrote:

The point of free software licenses, though (as opposed to proprietary
licenses), is not mainly to go to court (to “protect IP”, as the PSF
says - quite naively in my opinion); it is to enable trust among people.


Yes, that is true. Open source licenses are social documents as much as 
they are legal documents. However, they need to be legally enforceable 
so as to have their intended social effect.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Licensing // PSF // Motion of non-confidence

[VanL]

On 7/5/2010 8:03 PM, Steve Holden wrote:

Neil Hodgson wrote:

There have been moves in the past to simplify the license of Python
but this would require agreement from the current rights owners
including CWI and CNRI. IIRC not all of the rights owners are willing
to agree to a change.


That is the current position.


This is a pet project of mine, but it needs round tuits that are 
currently in short supply.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] PyCon 2011 Call for Tutorials

[VanL]

NB: Posting on behalf of Greg Lindstrom, our tutorial coordinator:

PyCon 2011 will be held March 9th through the 17th, 2011 in Atlanta,
Georgia. (Home of some of the best southern food you can possibly find
on Earth!) The PyCon conference days will be March 11-13, preceded by
two tutorial days (March 9-10), and followed by four days of development
sprints (March 14-17).

The call for tutorial proposals is open until November 1 and we want to
encourage you to submit a class idea!  Teachers are paid for their
efforts and it's a great opportunity for you to teach to a room full of
people who have paid to hear what you have to say?

Tutorials are 3-hour long classes (with a refreshment break) taught be
some of the leading minds in the Python community.  Classes range from
beginner (Introduction to Python) to advanced (OOP, Data Storage and
Optimization) and everything in between.  If you don't feel up to
teaching a class, you can always encourage your favorite mentor to teach
it!  Anything Python may be proposed for a class and a variety of topics
is always presented but we can't offer what isn't proposed!
Get more information at http://us.pycon.org/2011/about/ under the
"Tutorial Days" section.  Submit your idea on the site under the
"Speakers" tab.  That's it!  You could be teaching a class at PyCon next
March.  See you in Atlanta!


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Looking for an up-to-date address for john.benediktsson

[VanL]

Brett Cannon wrote:
> I was going through our list of email addreses for committers and
> realized that I don't have an address for john.benediktsson that
> works; his ewtllc.com address is bouncing. Anyone have an up-to-date
> one?

Replied privately.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.

[VanL]

Sigurd Torkel Meldgaard wrote:
> Hi to all Python developers
> 
> For a student project in a course on virtual machines, we are
> evaluating the possibility to
> experiment with removing the GIL from CPython...

Just an FYI, these two particular students already introduced themselves
on the PyPy list. Paolo is a masters student with experience in the
Linux kernel; Sigurd is a PhD candidate.

Their professor is Lars Bak, the lead architect of the Google V8
Javascript engine. They spent some time working on V8 in the last couple
months.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Fwd: Removal of GIL through refcounting removal.

[VanL]

VanL wrote:

> Just an FYI, 

...and the FYI was to no one in particular, just interesting information
from the PyPy list. It is just unfortunate timing that made it look like
a cheeky reply to Guido.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] GIL removal question

[VanL]

On 8/11/2011 2:11 PM, Sturla Molden wrote:


(b) another threading model (e.g. one interpreter per thread, as in Tcl,
Erlang, or .NET app domains).


We are close to this, in that we already have baked-in support for 
subinterpreters. Out of curiosity, why isn't this being pursued?


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] draft PEP: virtual environments

[VanL]

For what its worth

On 11/1/2011 11:43 AM, Paul Moore wrote:

On 1 November 2011 16:40, Paul Moore  wrote:

On 1 November 2011 16:29, Paul Moore  wrote:

On 31 October 2011 20:10, Carl Meyer  wrote:

For Windows, can you point me at the nt scripts? If they aren't too
complex, I'd be willing to port to Powershell.


For what its worth, there have been a number of efforts in this direction:

https://bitbucket.org/guillermooo/virtualenvwrapper-powershell
https://bitbucket.org/vanl/virtualenvwrapper-powershell

(Both different implementations)

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Python install layout and the PATH on win32

[VanL]

Following up on conversations at PyCon, I want to bring up one of my 
personal hobby horses for change in 3.3: Fix install layout on Windows, 
with a side order of making the PATH work better.


Short version:

1) The layout for the python root directory for all platforms should be 
as follows:


stdlib = {base/userbase}/lib/python{py_version_short}
platstdlib = {base/userbase}/lib/python{py_version_short}
purelib = {base/userbase}/lib/python{py_version_short}/site-packages
platlib = {base/userbase}/lib/python{py_version_short}/site-packages
include = {base/userbase}/include/python{py_version_short}
scripts = {base/userbase}/bin
data = {base/userbase}

2) On Windows, the python executable (python.exe) will be in the "bin" 
directory. That way the installer can optionally add just that directory 
to the PATH to pick up all python-related executables (like pip, 
easy_install, etc).


I have talked with a number of people at PyCon, including Tarek and MvL. 
Nobody objected, and several thought it was a good idea.


Long version:

As a bit of background,the layout for the Python root directory is 
different between platforms, varying in capitalization ("Include" vs. 
"include") and sometimes in the names of directories used ("Scripts" on 
Windows vs. "bin" most everywhere else). Further, the python version may 
or may not be included in the path to the standard library or not.


In times past, this layout was driven by an INSTALL_SCHEMES dict deep in 
the guts of distutils, but with distutils2 it has been lifted out and 
placed within a config file (sysconfig.cfg). [1]


Proposal #1 above deals with this inconsistency [1]. More concretely, it 
also makes it so that I can check in an entire environment into source 
control and work on it cross platform.


As an additional wrinkle on Windows, the main python binary (python.exe) 
is placed in the python root directory, but all associated runnable 
files are placed in the "Scripts" directory, so that someone who wants 
to run both Python and a Python script needs to add both $PYTHON_ROOT 
and $PYTHON_ROOT/Scripts to the PATH.


To add just a little more complication, the python binary is placed 
within the binaries directory when a virtualenv is created, leading to 
an inconsistency between regular python installs and virtualenvs.


Proposal #2 again provides consistency between virtualenvs and regular 
Python installs, and on windows it allows a single directory to be 
placed on the PATH to get all python-related binaries to run.


[1] 
https://bitbucket.org/tarek/distutils2/src/6c3d67ed3adb/distutils2/_backport/sysconfig.cfg

[2] It may be a foolish consistency, but I have a little mind.

Thanks, Van


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/13/2012 4:19 PM, Terry Reedy wrote:

What is {base/userbase} actually on a typical machine? It is fixed or
user choice?



It is based upon user choice and on whether it is a system-wide install 
(base) or a single-user install (userbase). Typically, though, it is 
just "where you installed Python" (/usr/local, C:\python\3.3, whatever).





___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/13/2012 3:11 PM, Brian Curtin wrote:


I'm familiar with the scripts/bin change. I take it the rest of that
stuff matches *nix? Text later on seems to support this, so I think
I'm on board with it.


Yes, that is correct.


Martin and I spoke on Friday and at least the bin/ folder and Path
stuff are acceptable and I'm working on the code for those.


[...]


If that virtualenv PEP is also accepted for 3.3, I think we can take
care of inconsistencies there (at least moving forward).


Thanks Brian!

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On Mar 13, 2012, at 8:37 PM, "Martin v. Löwis"  wrote:

>> 1) The layout for the python root directory for all platforms should be
>> as follows:
>> 
>> stdlib = {base/userbase}/lib/python{py_version_short}
>> platstdlib = {base/userbase}/lib/python{py_version_short}
>> purelib = {base/userbase}/lib/python{py_version_short}/site-packages
>> platlib = {base/userbase}/lib/python{py_version_short}/site-packages
>> include = {base/userbase}/include/python{py_version_short}
>> scripts = {base/userbase}/bin
>> data = {base/userbase}
> [...]
>> I have talked with a number of people at PyCon, including Tarek and MvL.
>> Nobody objected, and several thought it was a good idea.
> 
> I admit that I didn't understand that lib/python{version} was
> also part of the proposal. I'm fine with the bin/ change, but
> skeptical about the lib change - this just adds a redundant level
> of directories on Windows. The installation will end up in
> 
> c:\python33\lib\python3.3
> 
> which has the software name and version twice in the path.
> 
> Do we *really* need this?


We *already* have this. The only difference in this proposal is that we go from 
py_version_nodot to py_version_short, i.e. from

c:\python33\lib\python33 

to

c:\python33\lib\python3.3

Given that we already repeat it, isn't it better to be consistent?

Thanks,

Van

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/13/2012 9:58 PM, Terry Reedy wrote:


Given that we already repeat it, isn't it better to be consistent?


But there is no repetition currently on Windows installations.
I though you were just proposing to switch lib (lower-cased, and scripts
renamed as bin, and pythonxx). So I do not think I yet understand what
the proposal is and how it would be different from what I have now.


Aaah, I was looking at my local installations, which happen to be 
"nt-user". Looking at the system installation ("nt") I see that there is 
no repetition.


I am fine with keeping the distinction between base installs (no 
py_version) and user installs (including a py_version). I would just 
suggest that when you have a py_version, it be the same py_version (not 
dots sometimes, nodot other times).


It also begs the question as to whether the py_version is *ever* needed.

Thanks,
Van

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/14/2012 9:53 AM, Jim J. Jewett wrote:

In 
view-source:http://mail.python.org/pipermail/python-dev/2012-March/117586.html
van.lindberg at gmail.com posted:


1) The layout for the python root directory for all platforms should be
as follows:



stdlib = {base/userbase}/lib/python{py_version_short}
platstdlib = {base/userbase}/lib/python{py_version_short}
purelib = {base/userbase}/lib/python{py_version_short}/site-packages
platlib = {base/userbase}/lib/python{py_version_short}/site-packages
include = {base/userbase}/include/python{py_version_short}
scripts = {base/userbase}/bin
data = {base/userbase}


Why?

Pure python vs compiled C doesn't need to be separated at the directory
level, except for cleanliness.



I am deliberately being cautious here. I actually agree with you. I am 
only suggesting we maintain all of these different distinctions because 
that is what we have already.


You can see what we have currently at 
http://hg.python.org/distutils2/file/2cec52b682a9/distutils2/_backport/sysconfig.cfg


I am *not* suggesting that docs, etc change at all - that is included in 
a different part of the configuration and is not modified by what I 
propose here (lines 1-26).


As noted earlier in the thread, I also change my proposal to maintain 
the existing differences between system installs and user installs.


Thus, the only place I am proposing changing are the values for the keys 
listed above. Specifically, this (lines 57-65 in the file above):


[nt]
stdlib = {base}/Lib
platstdlib = {base}/Lib
purelib = {base}/Lib/site-packages
platlib = {base}/Lib/site-packages
include = {base}/Include
platinclude = {base}/Include
scripts = {base}/Scripts
data = {base}

Would become this:

[nt]
stdlib = {base}/lib
platstdlib = {base}/lib
purelib = {base}/lib/site-packages
platlib = {base}/lib/site-packages
include = {base}/include
platinclude = {base}/include
scripts = {base}/bin
data = {base}

and this (lines 86-93):

[nt_user]
stdlib = {userbase}/Python{py_version_nodot}
platstdlib = {userbase}/Python{py_version_nodot}
purelib = {userbase}/Python{py_version_nodot}/site-packages
platlib = {userbase}/Python{py_version_nodot}/site-packages
include = {userbase}/Python{py_version_nodot}/Include
scripts = {userbase}/Scripts
data = {userbase}

would become this:

[nt_user]
stdlib = {userbase}/python{py_version_short}
platstdlib = {userbase}/python{py_version_short}
purelib = {userbase}/python{py_version_nodot}/site-packages
platlib = {userbase}/python{py_version_nodot}/site-packages
include = {userbase}/python{py_version_nodot}/include
scripts = {userbase}/bin
data = {userbase}



... if you're rewriting that logic, you're just asking for bugs on a
strange platform that you don't use.


I am not rewriting the logic - the logic is driven by these 
configuration values. And this is a platform I use, and that is why this 
drives me crazy!




Subdirectories are:


You forgot one:

C:\python32\Scripts
Would change to C:\python32\bin. The python binary and scripts meant to 
be run direction (easy_install, etc) would all go in this directory.



C:\python32\DLLs

Would not change.


C:\python32\Doc

Would not change.


C:\python32\include

Would be specified as lower case only - but otherwise would not change.



C:\python32\Lib

Would be specified as lower case only - but otherwise would not change.


C:\python32\libs

Would not change.


C:\python32\tcl

Would not change.


C:\python32\Tools
This proposal does not change this, although I do think that this could 
be eliminated or made into "examples".


Thanks,
Van


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/14/2012 10:09 AM, Scott Dial wrote:

I think you are confusing two different configuration sections in
sysconfig.cfg:

[nt]
stdlib = {base}/Lib
platstdlib = {base}/Lib
purelib = {base}/Lib/site-packages
platlib = {base}/Lib/site-packages
include = {base}/Include
platinclude = {base}/Include
scripts = {base}/Scripts
data = {base}

[nt_user]
stdlib = {userbase}/Python{py_version_nodot}
platstdlib = {userbase}/Python{py_version_nodot}
purelib = {userbase}/Python{py_version_nodot}/site-packages
platlib = {userbase}/Python{py_version_nodot}/site-packages
include = {userbase}/Python{py_version_nodot}/Include
scripts = {userbase}/Scripts
data = {userbase}



I was lumping them together, yes, but now note that I modify the 
proposal to maintain this distinction. These would change to:


[nt]
stdlib = {base}/lib
platstdlib = {base}/lib
purelib = {base}/lib/site-packages
platlib = {base}/lib/site-packages
include = {base}/include
platinclude = {base}/include
scripts = {base}/bin
data = {base}

[nt_user]
stdlib = {userbase}/python{py_version_short}
platstdlib = {userbase}/python{py_version_short}
purelib = {userbase}/python{py_version_short}/site-packages
platlib = {userbase}/python{py_version_short}/site-packages
include = {userbase}/python{py_version_short}/include
scripts = {userbase}/bin
data = {userbase}


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/14/2012 1:32 AM, Mark Hammond wrote:

As per comments later in the thread, I'm -1 on including
"python{py_version_short}" in the lib directories for a number of
reasons; one further reason not outlined is that it would potentially
make running Python directly from a built tree difficult. For the same
reason, I'm also -1 on having that in the include dir.


A built tree would look the same as always - the directories would only 
be moved (if at all) during installation. Thus, you will still be able 
to run python directly from a built installation.


Also note that the py_version_short will not be in platform installs.



scripts = {base/userbase}/bin


We should note that this may cause pain for a number of projects - I've
seen quite a few projects that already assume "Scripts" on Windows - eg,
virtualenv and setuptools IIRC


If you look at these projects, though, they *special case* Windows to 
account for the different layout. Removing this difference will allow 
these projects to remove their special-casing code.




- and also assume the executable is where
it currently lives - one example off the top of my head is the mozilla
"jetpack" project - see:

https://github.com/mozilla/addon-sdk/blob/master/bin/activate.bat#L117


This code actually reinforces my point:

First, this code would actually still work. The section ":FoundPython" 
sets the PATH to "%VIRTUAL_ENV%\bin;%PYTHONINSTALL%;%PATH%" (L80), which 
would still allow python.exe to be found and run.


Second, look at that line again. Mozilla actually has edited this code 
so that the jetpack uses a cross-platform "bin" convention, just as I am 
proposing.


Third, one element of this proposal is that there would be a key placed 
in the registry that points directly to the location of the python 
executable, making locating it trivial to locate programmatically on 
Windows.


Thanks,
Van



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/14/2012 10:56 AM, Terry Reedy wrote:

Are you talking about 'install for all users' versus 'install for this
user only'? I have always done the former as I see no point to the
latter on my machine, even if another family member has an account.


Yes, but some people are on corporate machines that only allow "install 
for this user" installations.




I am fine with keeping the distinction between

 > base installs (no py_version)

I have no idea what this means. As far as I can remember, each
installation of Python x.y (back to 1.3 for me, on DOS) has gone into a
pythonxy (no dot) directory, with subdirectories much as Jim J. described.


I am referring to the currently-existing install schemes 'nt' ('install 
for all users') and 'nt-user' ('install for this user only'). The 
*current* layouts are described at 
http://hg.python.org/distutils2/file/2cec52b682a9/distutils2/_backport/sysconfig.cfg:


L57-65:
[nt]
stdlib = {base}/Lib
platstdlib = {base}/Lib
purelib = {base}/Lib/site-packages
platlib = {base}/Lib/site-packages
include = {base}/Include
platinclude = {base}/Include
scripts = {base}/Scripts
data = {base}

L86-93:
[nt_user]
stdlib = {userbase}/Python{py_version_nodot}
platstdlib = {userbase}/Python{py_version_nodot}
purelib = {userbase}/Python{py_version_nodot}/site-packages
platlib = {userbase}/Python{py_version_nodot}/site-packages
include = {userbase}/Python{py_version_nodot}/Include
scripts = {userbase}/Scripts
data = {userbase}

I am proposing that these change to:

[nt]
stdlib = {base}/lib
platstdlib = {base}/lib
purelib = {base}/lib/site-packages
platlib = {base}/lib/site-packages
include = {base}/include
platinclude = {base}/include
scripts = {base}/bin
data = {base}

[nt_user]
stdlib = {userbase}/python{py_version_short}
platstdlib = {userbase}/python{py_version_short}
purelib = {userbase}/python{py_version_short}/site-packages
platlib = {userbase}/python{py_version_short}/site-packages
include = {userbase}/python{py_version_short}/include
scripts = {userbase}/bin
data = {userbase}

All the other diuectories that Jim talked about would not be affected by 
this proposal.


Does this make it clearer?

Thanks,
Van

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/14/2012 5:39 PM, Mark Hammond wrote:

Can you offer any examples of 3rd party tools which could unify code in
this scheme, and particularly, where this scheme would cause them to
have less code, not more?


How about virtualenv:

"""
def path_locations(home_dir):
  """Return the path locations for the environment (where libraries are,
  where scripts go, etc)"""
  # XXX: We'd use distutils.sysconfig.get_python_inc/lib but its
  # prefix arg is broken: http://bugs.python.org/issue3386
  if sys.platform == 'win32':
[snip a bit about spaces in the path]
lib_dir = join(home_dir, 'Lib')
inc_dir = join(home_dir, 'Include')
bin_dir = join(home_dir, 'Scripts')
  if is_jython:
lib_dir = join(home_dir, 'Lib')
inc_dir = join(home_dir, 'Include')
bin_dir = join(home_dir, 'bin')
  elif is_pypy:
lib_dir = home_dir
inc_dir = join(home_dir, 'include')
bin_dir = join(home_dir, 'bin')
  elif sys.platform != 'win32':
lib_dir = join(home_dir, 'lib', py_version)
inc_dir = join(home_dir, 'include', py_version + abiflags)
bin_dir = join(home_dir, 'bin')
  return home_dir, lib_dir, inc_dir, bin_dir
"""



I think you misunderstand the .bat file - there is no python executable
in the bin directory. The bat file is locating your already installed
Python and attempting to use it.


My only point here is that it would still find the already-installed 
Python (I think).



My other questions still remain: who specifically will benefit from
this, and what would be the cost to those beneficiaries of sticking with
the existing scheme?


I will benefit, for one. My use case is that I do cross-platform 
development and deployment, and I occasionally want to put an entire 
environment in source control. Currently the case changing and 
Scripts/bin distinction make this a distinct pain, such that I go in and 
edit my Windows python installation in the way that I am describing 
right now.


From my actual experience with this layout, pip, virtualenv, and pypm 
are the only three major packages that hard-code this logic and would 
need to be changed slightly.


Thanks,
Van

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/14/2012 6:30 PM, Mark Hammond wrote:


So why not just standardize on that new layout for virtualenvs?


That sounds like the worst of all worlds - keep all the existing special 
cases, and add one.


The fact is that most code doesn't know about this, only installers or 
virtual environments. Most code just assumes that distutils does its 
thing correctly and that binaries are installed... wherever the binaries go.


Again, I have experience with this, as I have edited my own install to 
do this for a couple of years. The breakage is minimal and it makes 
things much more consistent and easier to use for cross-platform 
development.


Right now we are in front of the knee on major 3.x adoption - I would 
like to have things be standardized going forth everywhere.


Thanks,
Van

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/15/2012 6:19 PM, Mark Hammond wrote:

[At least I think that is your proposal - can you confirm that the
directory layouts in your proposal exactly match the directory
layouts in virtual envs on all other platforms? ie, that
inconsistencies like the python{py_version_short} suffix will not
remain?]


Yes, that is the ideal.


Also, I'm still yet to see what exactly becomes "easier" in your
model? As you mention, most Python code will not care; distutils and
other parts of the stdlib will "do the right thing" - and indeed,
already do for Windows.



Again, I have stated my use case - cross platform development where the
tools use the directory layout in some way, or where the environment
should be checked into source control.


It is a shame this wasn't done as part of py3k in the first place.
But I assume you would be looking at Python 3.4 for this, right?


No, I would like this for 3.3.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

Germane to this discussion, I reached out for feedback. Most people 
didn't care about the issue, or were slightly inclined to have it be 
uniform across platforms.


As Terry mentioned, I think that long-term uniformity will benefit 
everybody down the line, and that is the way to go.


The most interesting feedback, though, related to moving the Python exe 
and placing it on the PATH. I got one argument back that I thought was 
persuasive here: We want things to 'just work.' Specifically, the 
following sequence of events should not require any fiddling on Windows:


1. Install python.
2. Open up a shell and run "python"
3. Use pip or easy_install to install regetron (a package that installs 
an executable file).

4. Run regetron.

For step #2, the python exe needs to be on the PATH. For steps 3 and 4, 
the binaries directory needs to be on the PATH.


In hearing from a couple people who teach python to beginners, this is a 
substantial hurdle - the first thing they need to do is to edit their 
environment to add these directories to the PATH.


This is orthogonal to the Scripts/bin issue, but I thought it should be 
brought up.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/20/2012 10:52 AM, "Martin v. Löwis" wrote:

In hearing from a couple people who teach python to beginners, this is a
substantial hurdle - the first thing they need to do is to edit their
environment to add these directories to the PATH.

This is something I never understood. On Windows, it's custom to launch
programs from the start menu, and Python is easy enough to find on the
start menu (e.g. by typing "Python"). Why do people want to launch it by
opening a shell window, then typing python?


Because the workflow you suggest is broken when you are developing with 
Python. Assume that you are iteratively building up a program in Python. 
You aren't sure if it is right yet, so you want to get it into python to 
test it and see the output. There are three ways to do this.


1. Run python from the start menu.
- Import sys, fiddle with sys.path to add my module, import/run my 
module, do my tests. When you exit /hard error out, the python window 
disappears.


2. Double-click the .py file
- Runs the file, but then disappears immediately (unless you put in 
something like input/raw_input just to keep the window open) - and if it 
errors out, you never see the traceback - it disappears too fast.


3. Get a shell and run python.
This requires cd'ing to the directory where my .py file is, but then I 
run/import it and I see the information. To repeat the process, either 
type python again or just press arrow-up.


4. (Not relevant here) - do it in an IDE that does #3 for you.

#3 is the only reasonable way to do development if you are not in an IDE.

Thanks,
Van
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/20/2012 11:19 AM, "Martin v. Löwis" wrote:
No - there is an version #3a: 3.a) Get a shell and run the script CD 
into the directory, then directly run foo.py, without prefixing it 
with python.exe. This doesn't require any changes to the path, and is 
shorter in usage than having the path specified.


Fair enough - but notice: 1) You are still in the shell, instead of 
running from the start menu; and 2) what if you want to import it and 
test a function with various inputs? You either implement a 
request/response in a __main__ block, or type "python" and then import foo.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/20/2012 11:19 AM, "Martin v. Löwis" wrote:
No - there is an version #3a: 3.a) Get a shell and run the script CD 
into the directory, then directly run foo.py, without prefixing it 
with python.exe. This doesn't require any changes to the path, and is 
shorter in usage than having the path specified. With PEP 397, you 
will be able to run "py foo.py" without path modification, and it will 
get the correct Python version even (which neither the path 
manipulation nor the file association could achieve). 


There is also one more scenario, assuming that your project includes 
other libraries. You can run setup.py directly in your example, but what 
about pip or easy_install? Both of those require the binaries directory 
to also be on the PATH - requiring at least a little PATH manipulation.


Thanks,
Van
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/20/2012 1:50 PM, Merlijn van Deen wrote:

On 13 March 2012 20:43, VanL  wrote:

Following up on conversations at PyCon, I want to bring up one of my
personal hobby horses for change in 3.3: Fix install layout on Windows, with
a side order of making the PATH work better.


As this is being considered an 'incompatible change' on the bug
tracker item [1] in any case, I'd like to mention that this might also
be a convenient moment to re-think the default install location. After
all, software is supposed to be installed in %programfiles% on
windows, not in c:\.


I don't particularly care about this issue, as I always install to my 
own location (c:\lib\python\X.Y), but I don't think that the default 
location of the install should be confounded with this issue - or should 
I say these issues, because we already have two.



___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/20/2012 1:56 PM, Paul Moore wrote:

 This is covered (better, IMO) by PEP 397 - Python Launcher for
 Windows. Step 2, just run "py". If you prefer a particular version,
 run "py -2" or "py -3" or "py -3.2".


Interesting. I haven't played around with that at all, so can't comment. 
I will have to try it out.



 Adding python.exe to PATH actually makes this message *worse* as it
 confuses the issue. In a post-PEP 397 world, I would say that we
 should be telling Windows users *not* to run python.exe at all.
 (Unless they are using virtualenvs - maybe PEP 397 could do with an
 extra option to run the Python from the currently active virtualenv,
 but that's a side issue).


I think that having the PATH manipulation be optional might address this 
issue. I also think that the PEP 397 launcher should respect virtualenvs 
- at least the built-in pyvenvs - or else there will be lots of confusion.




 For your steps 3 and 4, there is certainly user intervention
 required as things stand. It would indeed be nice if "regetron" just
 worked as expected. But I'd argue a few points here:

 1. On Windows, if regetron was not explicitly an application for
 working with my Python installation (like pip, easy_install, nose,
 etc) then I'd prefer it to be packaged as a standalone application
 using cx_Freeze or py2exe. I've had too many "applications" break
 because I accidentally uninstalled a dependency from my Python
 installation to want anything that is an end-user application
 installed into my Python scripts/bin directory.


Maybe so - and I would probably agree that for any packaged application, 
bundling it into its own environment (or at least its own virtualenv) is 
the best practice. But what about pip, easy_install, nose, cython, 
pygments, PIL, etc, that do this and are meant to be associated with a 
particular python version? Substitute "nose" for "regetron" if you want, 
and there is still a problem.



 The problem with your example is that it depends on the
 package/executable. I looked at regetron (I thought it was a made up
 example at first!)


...! I got the name from the feedback I received. I thought it was made 
up too.



 I have no idea what proportion of Python users on Windows have
 multiple versions installed. I also have no idea how many such users
 work on the command line. My guess would be "not that many" and "not
 that many of the first group" respectively... But there are big
 groups like scientists and web developers who could sway these
 figures a lot.


There are a number of casual users that probably only have one version 
installed, but every python user/dev on windows that I know has one 
python that they consider to be "python," and everything else needs to 
be launched with a suffix (e.g., python26.exe). This is usually put 
earlier on the PATH so that it gets picked up first. For example, right 
now I have 2.6, 2.7, 3.2, jython, and pypy all installed, and I have 
"python" pointing to 2.7.



 I'd be curious as to how much PEP 397's py.exe would have helped
 those people. But yes, it's an issue. Although someone at some point
 will have to introduce those beginners to the question of Python 2 vs
 Python 3, and PATH pain will hit them then, anyway.


I would imagine that it would help steps 1 and 2, but 3 and 4 would be 
problematic (how can you pip install something using py?) unless you 
were in a virtualenv, and then (unless py respected the virtualenv) the 
whole thing would be problematic, because there wouldn't be one clear 
way to do it.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/20/2012 3:15 PM, Paul Moore wrote:

On 20 March 2012 19:35, Lindberg, Van  wrote:

I would like to know if you would object to user lib installs matching
the system install. I.e., would it cause problems with you if it were
just 'lib' everywhere, with no 'lib/python{version}'? It sounded like
adding the version directory was the issue.


User lib installs don't bother me as I don't use them. But yes, it's
the version directory that bothers me.

So if you're proposing simply making the user lib install match the
system install, both being lust "lib", then that's fine. I was
somewhat confused about what you were proposing, that's all.


I was originally going to make it match posix-user installs, but just 
plain posix (no version directory) is just fine with me too.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On 3/20/2012 4:49 PM, Mark Hammond wrote:


So I'm assuming that:
* The executable (and DLL) are moved to a "bin" directory in an 
installed Python.
* distutils etc will change to install all "scripts" (or executables 
generated from scripts) into that same directory.  IOW, "Scripts" 
would die.
* A virtual-env would have an almost identical layout to an installed 
Python.


Yes. I would make your point #3 stronger - I would like a virtualenv to 
have an identical layout to an installed python, at least with reference 
to the names of directories and the location of binaries. The base 
directory would be the only difference.


Thanks,
Van
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

On Tuesday, March 20, 2012 at 5:07 PM, Paul Moore wrote:
> It's worth remembering Éric's point - distutils is frozen and changes
> are in theory not allowed. This part of the proposal is not possible
> without an exception to that ruling. Personally, I don't see how
> making this change could be a problem, but I'm definitely not an
> expert.
>  
> If distutils doesn't change, bdist_wininst installers built using
> distutils rather than packaging will do the wrong thing with regard to
> this change. End users won't be able to tell how an installer has been
> built.
>  
>  


This is a good point. Who can make this call - Guido, or someone else?  

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)

[VanL]

As this has been brought up a couple times in this subthread, I figured 
that I would lay out the rationale here.


There are two proposals on the table: 1) Regularize the install layout, 
and 2) move the python binary to the binaries directory. This email will 
deal with the first, and a second email will deal with the second.


1) Regularizing the install layout:

One of Python's strengths is its cross-platform appeal. Carefully-
written Python programs are frequently portable between operating
systems and Python implementations with very few changes. Over the
years, substantial effort has been put into maintaining platform
parity and providing consistent interfaces to available functionality,
even when different underlying implementations are necessary (such
as with ntpath and posixpath).

One place where Python is unnecessarily different, however, is in
the layout and organization of the Python environment. This is most
visible in the name of the directory for binaries on the Windows 
platform ("Scripts") versus the name of the directory for binaries on 
every other platform ("bin"), but a full listing of the layouts shows

substantial differences in layout and capitalization across platforms.
Sometimes the include is capitalized ("Include"), and sometimes not; and
the python version may or may not be included in the path to the
standard library or not.

This may seem like a harmless inconsistency, and if that were all it 
was, I wouldn't care. (That said, cross-platform consistency is its own 
good). But it becomes a real pain when combined with tools like 
virtualenv or the new pyvenv to create cross-platform development 
environments.


In particular, I regularly do development on both Windows and a Mac, and 
then deploy on Linux. I do this in virtualenvs, so that I have a 
controlled and regular environment. I keep them in sync using source 
control.


The problem comes when I have executable scripts that I want to include 
in my dvcs - I can't have it in the obvious place - the binaries 
directory - because *the name of the directory changes when you move 
between platforms.* More concretely, I can't hg add "Scripts/runner.py? 
on my windows environment (where it is put in the PATH by virtualenv) 
and thendo a pull on Mac or Linux and have it end up properly in 
"bin/runner.py" which is the correct PATH for those platforms.


This applies anytime there are executable scripts that you want to 
manage using source control across platforms. Django projects regularly 
have these, and I suspect we will be seeing more of this with the new 
"project" support in virtualenvwrapper.


While a few people have wondered why I would want this -- hopefully 
answered above -- I have not heard any opposition to this part of the 
proposal.


This first proposal is just to make the names of the directories match 
across platforms. There are six keys defined in the installer files 
(sysconfig.cfg and distutils.command.install): 'stdlib', 'purelib', 
'platlib', 'headers', 'scripts',  and 'data'.


Currently on Windows, there are two different layouts defined:

  'nt': {
'stdlib': '{base}/Lib',
'platstdlib': '{base}/Lib',
'purelib': '{base}/Lib/site-packages',
'platlib': '{base}/Lib/site-packages',
'include': '{base}/Include',
'platinclude': '{base}/Include',
'scripts': '{base}/Scripts',
'data'   : '{base}',
},

  'nt_user': {
'stdlib': '{userbase}/Python{py_version_nodot}',
'platstdlib': '{userbase}/Python{py_version_nodot}',
'purelib': '{userbase}/Python{py_version_nodot}/site-packages',
'platlib': '{userbase}/Python{py_version_nodot}/site-packages',
'include': '{userbase}/Python{py_version_nodot}/Include',
'scripts': '{userbase}/Scripts',
'data'   : '{userbase}',
},


The proposal is to make all the layouts change to:

  'nt': {
'stdlib': '{base}/lib',
'platstdlib': '{base}/lib',
'purelib': '{base}/lib/site-packages',
'platlib': '{base}/lib/site-packages',
'include': '{base}/include',
'platinclude': '{base}/include',
'scripts': '{base}/bin',
'data'   : '{base}',
},

The change here is that 'Scripts' will change to 'bin' and the 
capitalization will be removed. Also, "user installs" of Python will 
have the same internal layout as "system installs" of Python. This will 
also, not coincidentally, match the install layout for posix, at least 
with regard to the 'bin', 'lib', and 'include' directories.


Again, I have not heard *anyone* objecting to this part of the proposal 
as it is laid out here. (Paul had a concern with the lib directory 
earlier, but he said he was ok with the above).


Please let me know if you have any problems or concerns with this part 1.

Thanks,
Van

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)

[VanL]

[PART 2: Moving the python binary]

There are two proposals on the table: 1) Regularize the install layout, 
and 2) move the python binary to the binaries directory. This email 
deals with the second issue exclusively. This has been the more

contentious issue.

2) Moving the Python exe:

A regular complaint of those new to Python on windows (and new to 
programming generally) has been that one of the first things that they 
need to do is to edit the PATH to allow Python to be run. In particular, 
this is the difficult sequence:


1. Install python.
2. Open up a shell and run "python"
3. Use pip or easy_install to install regetron (a package that installs 
an executable file).

4. Run regetron.

For step #2, the python exe needs to be on the PATH. For steps 3 and 4, 
the binaries directory needs to be on the PATH. Currently, neither of 
these are true, so the path needs to be edited to include both the 
python root (where python.exe is, for step 2) and the "Scripts" 
(hopefully soon "bin") directory where "pip" and "regetron" are (for 
steps 3 and 4). You can substitute "regetron" for "nose," "cython," or 
other packages as well.


MvL asked why anyone would want to run python directly from a cmd shell 
instead of running it from the start menu. There are two immediate 
responses to that: 1) observed behavior is that people prefer to run 
"python" from the cmd shell when developing (as observed by various 
people teaching Python, including Brian Curtin in this thread), and 2) 
running python or python programs from the shell is sometimes the only 
way to get a proper traceback when developing, making it a better way to 
work.


The proposal here is to move the python.exe into the binaries directory 
(whatever it is called) and add an option to the windows installer to 
add that one directory to the PATH on install (and clean up the PATH on 
uninstall). A new registry key would be added pointing to the location 
of the python binary (wherever it is).


Brian Curtin suggested this part of the proposal and has implemented it 
in a branch. MvL suggested a gradual transition to this over a 
three-release period.


Open Issues:

The PEP 397 Installer: As pointed out by Paul Moore, it may not matter 
once PEP 397 lands if python.exe is in the PATH or not - and it may be 
better if it is not. As he put it:


"""If we do put python.exe on PATH (whether it's in bin or not), we have
to debate how to handle people having multiple versions of python on
their machine. In a post-PEP 397 world, no Python is "the machine
default" - .py files are associated with py.exe, not python.exe, so we
have to consider the following 3 commands being run from a shell
prompt:

1. myprog.py
2. py myprog.py
3. python myprog.py

1 and 2 will always do the same thing. However, 3 could easily do
something completely different, if the Python in the #! line differs
from the one found on PATH. To me, this implies that it's better for
(3) to need explicit user action (setting PATH) if it's to do anything
other than give an error. But maybe that's just me. I've been hit too
often by confusion caused by *not* remembering this fact."""

One possible response here is that the moving of the python.exe binary 
and the setting of the PATH would be tied to an unchecked-by-default 
installer option, making an explicit user choice needed to invoke the 
new functionality.


Breakage of existing tools: Mark Hammond, Paul Moore, and Tim Golden 
have all expressed that they have existing tools that would break and 
would need to be adjusted to match the new location of the python.exe, 
because that location is assumed to be at the root of the python install.


A related issue is that this portion of the proposal has met with some 
resistance, but not much support here on Python-dev. The reason for that 
is selection bias: Those who are on Python-dev are much more likely to 
have tools that do advanced things with Python, such as introspect on 
the location of the binary, and are also much more likely to be 
comfortable with things like editing the PATH on windows. In contrast, 
the people that have trouble with this issue are those that are newest 
to Python and programming generally - those for whom editing the PATH is 
a challenge and whom are likely to be confused by the distinction 
between python.exe and a python program - and why, even after they add 
python to the path, the python program is not directly executable.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)

[VanL]

Hi Paul,

To start with, I appreciate your comments, and it is worth having both 
sides expressed.


On 3/22/2012 12:59 PM, Paul Moore wrote:
I'm repeating myself here after I promised not to. My apologies, but I 
don't think this posting captures the debate completely. One reason I 
suggested a PEP is to better ensure that the arguments both pro and 
con were captured, as that is a key part of the PEP process. 


I would be happy to write up a PEP.

First of all, this difference is almost entirely *invisible*. Apart
from possibly setting PATH (once!) users should not be digging around
in the Python installation directory. Certainly on Windows, it is a
very unusual application that expects users to even care how the
application is laid out internally. And I suspect that is also true on
Unix and Mac.


This is a good point; it is mostly visible in the virtualenvs. If it 
only changed in virtualenvs, I would be happy. The policy, though, is 
that the virtualenv follows the platform policy.



Secondly, the layouts are not as similar as you claim here, if I
understand things correctly. on Unix Python is installed in
/usr/local/bin so there isn't even a "Python installation directory"
there. And Macs use some sort of Mac-specific bundle technology as I
understand it. To be honest, I don't think that there's a lot of
similarity even with your proposed changes.


I was summarizing here because, frankly, there are hardly any OS/2 
users, so it would be mostly Windows users affected by this change. Also 
as noted, I suggest that all platforms standardize on bin, lib, and 
include, just as I laid out.


That said, while I think that the above is a good idea, my personal 
ambitions are more modest: If the names of the top-level directories 
only were changed to 'bin', 'lib', and 'include' - never mind 
differences under 'lib' - I would be happy. In fact, even the one change 
of 'Scripts' to 'bin' everywhere would get 90% of my uses.



The issue with virtualenv and pyvenv may be more significant. But
you're only mentioning those incidentally.


I am approaching it from the platform level because of the policy that 
virtualenvs match the platform install layout. If instead virtualenv 
layouts were standardized, that would end up making me just as happy.




(I also get a sense of undue haste -
"we can change the Python layout for 3.3, but changing packaging and
virtualenv is a much longer process"...)


Honestly, I didn't expect that much resistance. None of the people I 
talked to in person even cared, or if they did, they thought that 
consistency was a benefit. But now that virtualenvs are going in in 3.3, 
I see this as the last good chance to change this.




This presupposes that your development workflow - developing in place
in the virtualenv itself - is "the obvious approach". From what I've
seen of tools like virtualenvwrapper, you're not alone in this. [...] But can
you acknowledge (and document) that "change your way of working" is
another alternative to "change Python".



Acknowledged. What you say is true - but people wanted to know what the 
benefit would be. I laid out my concrete use-case as a rationale. And as 
you note, I am not alone in this type of development. Sure, I care here 
because it affects my style of development, and there are other styles 
that have other benefits (and tradeoffs). I don't see that this part of 
the proposal would negatively affect those styles.





While a few people have wondered why I would want this -- hopefully answered
above -- I have not heard any opposition to this part of the proposal.

See above. There has been opposition from a number of people. It's
relatively mild, simply because it's a niche area and doesn't cause
huge pain, but it's there. And you seem (based on the above analysis)
to be overstating the benefits, so the summary here is weighted
heavily in favour of change.


If I have misrepresented anyone, I am sorry - but to the best of my 
understanding, no one had (prior to you, right now) objected to *this 
part* of the proposal. Mark, at least, specified that his concern was 
with the moving of the python binary and that he didn't care about this 
part. I believe Tim indicated that too, but perhaps I have on my 
rose-colored glasses and misunderstood him.



Also, you have made no attempt that I've seen to address the question
of why this is important enough to break backward compatibility. Maybe
it is - but why? Backward compatibility is a very strong rule, and
should be broken only with good justification. Consistency, and "it
makes my way of working easier" really shouldn't be sufficient.


In general, yes, I agree with you. However, the break with backwards 
compatibility is, as you point out, minor, and there is a benefit to 
consistency - especially given virtualenv-centric development.




Has anyone checked whether this will affect people like Enthought and
ActiveState who distribute their own versions of Python? Is
ActiveState's PPM tool

Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 2: Moving the python.exe)

[VanL]

Another use case was just pointed out to me: making things consistent with 
buildout. Given a similar use case (create repeatable cross platform 
environments), they create and use a 'bin' directory for executable files. 
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)

[VanL]

On Friday, March 23, 2012 at 11:39 AM, PJ Eby wrote:
> Even if you are using tools that don't use distutils' configuration settings 
> for these directories, why not simply fix those tools so that they do?  

Thats what I do currently - I set things to bin and patch Python and the tools 
so that they work.

However, have considered this to be a little bit of a wart anyway for a long 
time - even before I adopted my current method of working - because it is a 
pointless inconsistency.

The fact that it makes virtual environments consistent across platforms, 
together with pyvenv going into 3.3, gave me enough of a push to elevate this 
from private annoyance to "should fix."

Thanks,
Van
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)

[VanL]

On Friday, March 23, 2012 at 1:35 PM, PJ Eby wrote: 
> Tool developers are going "meh" about your proposal because it doesn't 
> actually solve any problems for them: they still have to support the old 
> layout, and if their code already uses distutils' facilities for obtaining 
> paths, there's nothing they gain from the change.

Three notes. FIrst, distutils.cfg doesn't always work because it is centered 
around the idea of set paths that are the same each time - which doesn't always 
work with virtualenvs. Further, a number of tools find that it doesn't work 
(haven't seen it myself, but look at the comments in pypm's installer). So yes, 
I patch python.

Second, most installer tools don't follow distutils.cfg. Even if that helps for 
python setup.py install, the other tools are still broken when you want to 
specify a layout. That is why changing the defaults is the only effective way 
to make this change - because the defaults drive what is actually implemented. 
I know, because I have looked at and patched these tools to make them work.

Third, there are some tool makers going meh - because you are right, this is 
not a problem they have. This is a problem associated with using those tools. 
And regardless of there being other ways to do it, including your 'dump it in a 
directory' method, development in virtualenvs is convenient, widespread, and on 
the rise. Given that pyvenv will go into 3.3, it will be the 'one obvious way 
to do it' - making going-forward cross-platform compatibility a positive good. 
Again I note the example of buildout.

And fourth, (because nobody expects the spanish inquisition), isn't the 
gratuitous difference a (small but) obvious wart? Does anybody positively like 
'Scripts'? The most common comment I have received when talking to people off 
this list has been, 'yeah, that was always sort of weird.'___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32 (Rationale part 1: Regularizing the layout)

[VanL]

On Mar 23, 2012 10:21 PM, "PJ Eby"  wrote:
>
>
> On Mar 23, 2012 4:19 PM, "VanL"  wrote:
> >
> > Three notes. FIrst, distutils.cfg doesn't always work because it is
centered around the idea of set paths that are the same each time - which
doesn't always work with virtualenvs.
>
> And the virtualenv doesn't contain its own copy of distutils.cfg?

It can, but a new one. Virtualenvs don't carry over the distutils.cfg from
the main installation. Thus, using distutils.cfg in the virtualenv would
require editing the .cfg for every new virtualenv-and it still wouldn't
work all the time for the other reasons discussed.

> > Second, most installer tools don't follow distutils.cfg. Even if that
helps for python setup.py install, the other tools are still broken when
you want to specify a layout.
>
> So, we should change Python to fix the broken tools that don't follow
documented standards for configuring installation locations?

If the documented functions don't work for the use cases, there is nothing
else. Again, see the comments in PyPM.

>
> If the tools are that broken, aren't they going to break even *harder*
when you change the paths for Windows?

If people substitute on hard-coded value for another, does cross platform
consistency help? And ifthat focuses attention on the new packaging APIs
and the correct way to do it, isn't that even better?

> > And fourth, (because nobody expects the spanish inquisition), isn't the
gratuitous difference a (small but) obvious wart?
>
> It's hardly the only wart we keep around for backwards compatibility.  If
it's going to change, it needs a proper transition period at the least.

Already proposed, making a transition over three releases with it starting
as an off by default option in 3.3.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python install layout and the PATH on win32

[VanL]

I heard back from Enthought on this part of the proposal. They could 
accommodate this change.


1) The layout for the python root directory for all platforms should be
as follows:

stdlib = {base/userbase}/lib/
platstdlib = {base/userbase}/lib/
purelib = {base/userbase}/lib/site-packages
platlib = {base/userbase}/lib/site-packages
include = {base/userbase}/include/python{py_version_short}
scripts = {base/userbase}/bin
data = {base/userbase}
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The path module PEP

[VanL]

The path module has a great idea, but it does too much -- it conflates 
too many ideas into a single module.

It has methods for dealing with files (open, bytes, read, etc) as well 
as methods for dealing with a filesystem tree as a whole (relpath, 
abspath, etc).  Both of these ideas are tangentially related to paths, 
but really aren't in the same conceptual box.

Not too long ago, I had to build something loosely based upon the path 
module.  Instead of using it as-is, I broke it up into three modules:

Tree (filesystem interfaces)
Path (*just* path interfaces)
File (a generic filelike object)

Doing it this way had two benefits:

First, it put different concepts into different modules.  I note that 
some other virtual filesystem modules also maintedned a similar 
separation - probably for similar reasons.

Second, I was able to define an interface which could be used across 
remote systems -- e.g. I was able to have an FTPTree (built on the 
standard library ftplib) and SSHTree (built upon paramiko) as well as 
FileTree (a standard filesystem).  This isn't full-fledged interfaces - 
I just implemented common functionality in a class and then delegated to 
a ._ops class which passed through the necessary operations.  However, I 
was able to use the various trees and file-like objects interchangeably.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] The path module PEP

[VanL]

One other benefit that I neglected to put into the previous post - I was 
able to maintain separate cwd's for each tree.

An example of use:

Each tree has its own context, independent of the context of python:

 >>> local, local2 = fs.LocalTree(), fs.LocalTree()
 >>> local.pwd
'/home/targoz'
 >>> local2.pwd
'/home/targoz'
 >>> os.getcwd()
'/home/targoz'
 >>> local.chdir('..')
 >>> local2.chdir('www')
 >>> local.pwd
'/home'
 >>> local2.pwd
'/home/targoz/www'
 >>> os.getcwd()
'/home/targoz'

Remote trees have the same interface:

 >>> remote_login_data = {'username': 'targoz', 'password': 'my_pass', 
host': 'hostname.com'}
 >>> remote = fs.SSHTree(access=remote_login_data)
 >>> remote.pwd()
'/home/nportal'

Trees can interact, regardless of whether they are local or remote:

 >>> local2.listdir('files')
['myfile', 'otherfile.txt']
 >>> remote.listdir()
[]
 >>> localfile = local2.open(('targoz/myfile') # Opens a file-like object
 >>> remote.savefile(localfile, 'remote_name')
 >>> remote.listdir()
['myfile']

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com