Re: [Python-Dev] Python environment registration in the Windows Registry

[Alexander Walters]

...just when I thought I have solved the registry headaches I have been 
dealing with...


I am not saying this proposal will make the registry situation worse, 
but it may break my solution to the headaches Python's registry use 
causes with some non-standard module installers (and even the standard 
distutils exe installers, but that is being mitigated).  In the wild 
exist modules with their own EXE or MSI installers that check the 
registry for 'the system python'.  No matter how hard you hit them, they 
will only install to *that one python*.  You can imagine the sadist that 
builds such an installer would not be receptive to the concept of 
wheels.  So in order to force those modules to install to the python YOU 
want, dammit, you have to edit the registry.  I have (contributed to) a 
script that just sets whatever python it was last run with to the one 
true system python.  Works with virtualenvs too.This is not a terribly 
obscure script either, by the way.  It is in the first reply to the 
google search for "how to install exes in a virtualenv".


So here I am in a situation where some pythons use another registry 
entry.  I have no idea if this makes my life as a user harder. What does 
this kind of meddling do against tools trying to autodetect python, and 
finding an ever changing value?  Are we trying to guarantee that the 
keys used by the python.org installers or only ever actually used by 
standard CPython?


I know for PTVS manually adding a python environment to visual studio is 
trivial - you fill in three locations, and its done.  Just today I added 
a python environment to my system that was not autodetected.  It took 
under a minute and almost no effort to add it... so for that tool this 
adds very little benefit.  I do not know about other tools.


On a very personal note (like the rest of this wasn't my personal issues 
with possibly making my life slightly more difficult), I would much 
rather see python stop touching the registry all together - but I have 
no strong argument for that.


On 2/3/2016 00:15, Steve Dower wrote:
I was throwing around some ideas with colleagues about how we detect 
Python installations on Windows from within Visual Studio, and it came 
up that there are many Python distros that install into different 
locations but write the same registry entries. (I knew about this, of 
course, but this time I decided to do something.)


Apart from not being detected properly by all IDEs/tools/installers, 
non-standard distros that register themselves in the official keys may 
also mess with the default sys.path values. For example, at one point 
(possibly still true) if you installed both Canopy and Anaconda, you 
would break the first one because they tried to load the other's stdlib.


Other implementations have different structures or do not register 
themselves at all, which also makes it more complicated for tools to 
discover them.


So here is a rough proposal to standardise the registry keys that can 
be set on Windows in a way that (a) lets other installers besides the 
official ones have equal footing, (b) provides consistent search and 
resolution semantics for tools, and (c) includes slightly more rich 
metadata (such as display names and URLs). Presented in PEP-like form 
here, but if feedback suggests just putting it in the docs I'm okay 
with that too. It is fully backwards compatible with official releases 
of Python (at least back to 2.5, possibly further) and does not 
require modifications to Python or the official installer - it is 
purely codifying a superset of what we already do.


Any and all feedback welcomed, especially from the owners of other 
distros, Python implementations or tools on the list.


Cheers,
Steve

-

PEP: ???
Title: Python environment registration in the Windows Registry
Version: $Revision$
Last-Modified: $Date$
Author: Steve Dower 
Status: Draft
Type: ???
Content-Type: text/x-rst
Created: 02-Feb-2016


Abstract


When installed on Windows, the official Python installer creates a 
registry key for discovery and detection by other applications. 
Unofficial installers, such as those used by distributions, typically 
create identical keys for the same purpose. However, these may 
conflict with the official installer or other distributions.


This PEP defines a schema for the Python registry key to allow 
unofficial installers to separately register their installation, and 
to allow applications to detect and correctly display all Python 
environments on a user's machine. No implementation changes to Python 
are proposed with this PEP.


The schema matches the registry values that have been used by the 
official installer since at least Python 2.5, and the resolution 
behaviour matches the behaviour of the official Python releases.


Specification
=

We consider there to be a single collection of Python environments on 
a machine, where the collection may be different for each 

Re: [Python-Dev] Python environment registration in the Windows Registry

[Glenn Linderman]

On 2/3/2016 12:15 AM, Alexander Walters wrote:
On a very personal note (like the rest of this wasn't my personal 
issues with possibly making my life slightly more difficult), I would 
much rather see python stop touching the registry all together - but I 
have no strong argument for that. 


Me too. My opinions follow, some might call them arguments, strong or 
weak, some might call them foolishness.


I've been setting up a collection of tools and programs on Dropbox for 
far-flung project team members (users, not programmers) to share. It is 
nearly impossible to install a typical Windows program on Dropbox, 
because the installation information is partly in the installed 
directory structure, and partly in the registry. Exporting registry info 
from the machine that does the install to other machines is hard because 
other users have different paths to Dropbox. OK, for commercial 
software, installing to Dropbox probably violates some of the licensing, 
forcing each user to buy/install, but FOSS or in-house software doesn't 
have such restrictions: but still, most of it wants to use the registry, 
and Windows almost, but doesn't quite, force it to.


Portable software exists, but often is 3rd party hacks of popular FOSS 
rather than being directly supported by the FOSS development team.  
Python falls into this category. Happily, I recently found WinPython 
Zero, which hacks it (somehow) to work portably, and I've deployed it 
successfully on Dropbox. I'd rather Python were portable on its own, 
without hacks.


Portability requires not using the registry, so I agree with Alexander 
there.


Portability, as "Windows portable software" is generally defined, is 
focused on moving programs and data around on a flash drive, from one 
computer to another, and is focused on single-user, any (Windows) 
machine (with sufficient specs).


That doesn't quite fit the Dropbox environment. Most portable software 
reintroduces the idea of storing configuration information in the 
program folder, which is OK for "project" configuration info, done once, 
by one person, but not for "personal preferences".


The other thing Windows GUI lost is the concept of "current working 
directory", which hit me hard when I first started trying to set up 
project working areas on Dropbox. Many Windows programs want to run only 
one copy of themselves, in one Window, with one set of settings, and one 
"Start In" directory (which generally defaults to  the program 
directory, or sometimes to "My Documents".  This is why I went looking 
for a portable Python (and other portable things), and I finally 
realized that I was "fighting city hall" in trying to get an environment 
set up that was usable for the various teams (of users, not developers). 
Writeup at slashdot 
 for more details 
on the lack of a "current working directory" concept in Windows GUI 
programs.


The path to Dropbox folders is different for everyone, even the drive 
letter can be different.


So here are my preferences for CPython:

1) It would be best CPython itself were fully portable. That wherever 
you point the installer, it lives there, and if somehow you happen to 
execute it from that directory, it would use its invocation path as the 
basis for finding the rest of the installed pieces.


2) A script could be provided that sets the association for .py and the 
corresponding ftype to point to the python that executed the script... 
and which has a parameter to clear that back out, too. This would be to 
allow users to set a particular python for convenient script execution, 
since Windows does the association thing, instead of using #! lines.


3) A script could be provided (maybe the one Alexander referred to) that 
sets the registry so that the "apparently one true System Python" points 
to the python that executed the script... and which has a parameter to 
clear that back out, too.  This would be for compatible transition to 
the new registry-free Python versions for packages that have weird 
installers (such as Alexander alluded to). But with the registry-free 
Python available, those packages would hopefully see the future is 
registry free, and avoid requiring registry data to do installs.


4) A script could be provided to add the python that executed the script 
to the PATH, with an option to remove it.


5) A script could be provided to add the python that executed the script 
to the launcher .ini file, with an option to remove it.


6) A script could be provided to add the python that executed it to the 
Start menu, and/or Desktop icons, with an option to remove them.


Maybe scripts 2-6 are all the same one, with different options (or 
different invocation shortcuts for the clicky folk). Not everyone would 
probably need all the scripts, but none of them would be particularly 
large, such that they would be burdensome for others to ignore.


Such a collection of scripts would allow folks to 

Re: [Python-Dev] Python environment registration in the Windows Registry

[Paul Moore]

On 3 February 2016 at 09:50, Glenn Linderman  wrote:
> On 2/3/2016 12:15 AM, Alexander Walters wrote:
>
>> On a very personal note (like the rest of this wasn't my personal issues
>> with possibly making my life slightly more difficult), I would much rather
>> see python stop touching the registry all together - but I have no strong
>> argument for that.
>
> Me too. My opinions follow, some might call them arguments, strong or weak,
> some might call them foolishness.

I also would prefer that Python not use/need the registry. As far as I
know, in practical terms Python works just fine without the registry
entries being present, but I'd like it if that were formally supported
as an option for users. Obviously it means that tools that want to see
what Python versions are installed won't see such unregistered copies,
I'm fine with that (and I don't have a problem with leaving it up to
said tools as to whether they want to include ways to support using a
Python installation that's not in the registry).

Some issues with this proposal:

1. I don't like the way it states that Python distributions "must" set
keys. I'd rather that it were explicitly stated that a distribution
which sets no registry keys is entirely valid (with the obvious
implication that tools which scan the registry looking for installed
Python distributions won't be able to see such an installation).
2. It's not clear to me how alternative distributions will specify
their registry keys (or the fact that they don't use them). The
registry keys are built into the Python interpreter - this proposal
seems to imply that distributions that simply repackage the python.org
build will no longer be valid or supported, and instead anyone who
wants to produce a custom Python distribution (even private in-house
repackagings) will need to change the Python source and rebuild.
That's a pretty major change, and if that *is* the intent, then as a
minimum I'd say we need to provide compiler flags to let
redistributors specify their Company/Tag values (or say that they want
to omit registry use). And I'm still not happy that "repackage the
python.org binaries" has been removed as an option.

It's possible that the reason the above two points have been missed is
because the proposal focuses purely on "informational" registry data.
But Python also modifies sys.path based on the registry entries - and
possibly has other behavioural changes as well. The pywin32 package,
in particular, makes use of this (it's a feature of pywin32 that I
disagree with and I wish it didn't do that, but it does, and it's a
very widely used package on Windows). So ignoring this aspect of
Python's behaviour is a big problem. (Also, what changes will pywin32
need to make to correctly support being installed into non-python.org
distributions when this proposal is implemented?)

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Opcode cache in ceval loop

[francismb]

Hi,

On 02/01/2016 10:43 PM, Yury Selivanov wrote:

> 
> We also need to deoptimize the code to avoid having too many cache
> misses/pointless cache updates.  I found that, for instance, LOAD_ATTR
> is either super stable (hits 100% of times), or really unstable, so 20
> misses is, again, seems to be alright.
> 
Aren't those hits/misses a way to see how dynamic the code is? I mean
can't the current magic (manually tweaked on a limited set) values,
be self tweaked/adapted on those numbers?

Thanks in advance,
francis
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Opcode cache in ceval loop

[Sven R. Kunze]

On 03.02.2016 22:22, Yury Selivanov wrote:

One way of tackling this is to give each optimized opcode
a counter for hit/misses.  When we have a "hit" we increment
that counter, when it's a miss, we decrement it.


Within a given range, I suppose. Like:

c = min(c+1, 100)



I kind of have something like that right now:
https://github.com/1st1/cpython/blob/opcache5/Python/ceval.c#L3035

But I only decrement that counter -- the idea is that LOAD_ATTR
is allowed to "miss" only 20 times before getting deoptimized.

I'll experiment with inc/dec on hit/miss and see how that affects
the performance.

An ideal way would be to calculate a hit/miss ratio over time
for each cached opcode, but that would be an expensive
calculation.


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Opcode cache in ceval loop

[Yury Selivanov]

On 2016-02-03 3:53 PM, francismb wrote:

Hi,

On 02/01/2016 10:43 PM, Yury Selivanov wrote:


We also need to deoptimize the code to avoid having too many cache
misses/pointless cache updates.  I found that, for instance, LOAD_ATTR
is either super stable (hits 100% of times), or really unstable, so 20
misses is, again, seems to be alright.


Aren't those hits/misses a way to see how dynamic the code is? I mean
can't the current magic (manually tweaked on a limited set) values,
be self tweaked/adapted on those numbers?



Probably.

One way of tackling this is to give each optimized opcode
a counter for hit/misses.  When we have a "hit" we increment
that counter, when it's a miss, we decrement it.

I kind of have something like that right now:
https://github.com/1st1/cpython/blob/opcache5/Python/ceval.c#L3035

But I only decrement that counter -- the idea is that LOAD_ATTR
is allowed to "miss" only 20 times before getting deoptimized.

I'll experiment with inc/dec on hit/miss and see how that affects
the performance.

An ideal way would be to calculate a hit/miss ratio over time
for each cached opcode, but that would be an expensive
calculation.

Yury

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

> There is an old discussion about the performance of PyMem_Malloc() memory 
> allocator.

Oops, I forgot to mention that my patch is a follow-up of a previous
patch showing nice speedup on dict:
http://bugs.python.org/issue23601
(but I said it in my issue ;-))

Well, see http://bugs.python.org/issue26249 for the longer context.


2016-02-03 22:03 GMT+01:00 Victor Stinner :
> Does anyone recall the rationale to have two families to memory allocators?

I asked Mercurial, and I found the change addind PyMem_Malloc():
---
branch:  legacy-trunk
user:Guido van Rossum 
date:Tue Aug 05 01:59:22 1997 +
files:   Include/mymalloc.h
description:
Added Py_Malloc and friends as well as PyMem_Malloc and friends.
---

As expected, it's old, as the change adding PyObject_Malloc():
---
changeset:   12576:1c7c2dd1beb1
branch:  legacy-trunk
user:Guido van Rossum 
date:Wed May 03 23:44:39 2000 +
files:   Include/mymalloc.h Include/objimpl.h
Modules/_cursesmodule.c Modules/_sre.c Modules/_tkinter.c
Modules/almodule.c Modules/arraymodule.c Modules/bsddbmodule.
description:
Vladimir Marangozov's long-awaited malloc restructuring.
For more comments, read the patc...@python.org archives.
For documentation read the comments in mymalloc.h and objimpl.h.

(This is not exactly what Vladimir posted to the patches list; I've
made a few changes, and Vladimir sent me a fix in private email for a
problem that only occurs in debug mode.  I'm also holding back on his
change to main.c, which seems unnecessary to me.)
---

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Modify PyMem_Malloc to use pymalloc for performance

[Victor Stinner]

Hi,

There is an old discussion about the performance of PyMem_Malloc()
memory allocator. CPython is stressing a lot memory allocators. Last
time I made statistics, it was for the PEP 454:
"For example, the Python test suites calls malloc() , realloc() or
free() 270,000 times per second in average."
https://www.python.org/dev/peps/pep-0454/#log-calls-to-the-memory-allocator

I proposed a simple change: modify PyMem_Malloc() to use the pymalloc
allocator which is faster for allocation smaller than 512 bytes, or
fallback to malloc() (which is the current internal allocator of
PyMem_Malloc()).

This tiny change makes Python up to 6% faster on some specific (macro)
benchmarks, and it doesn't seem to make Python slower on any
benchmark:
http://bugs.python.org/issue26249#msg259445

Do you see any drawback of using pymalloc for PyMem_Malloc()?

Does anyone recall the rationale to have two families to memory allocators?

FYI Python has 3 families since 3.4: PyMem, PyObject but also PyMem_Raw!
https://www.python.org/dev/peps/pep-0445/

--

Since pymalloc is only used for small memory allocations, I understand
that small objects will not more be allocated on the heap memory, but
only in pymalloc arenas which are allocated by mmap. The advantage of
arenas is that it's possible to "punch holes" in the memory when a
whole arena is freed, whereas the heap memory has the famous
"fragmentation" issue because the heap is a single contiguous memory
block.

The libc malloc() uses mmap() for allocations larger than a threshold
which is now dynamic, and initialized to 128 kB or 256 kB by default
(I don't recall exactly the default value).

Is there a risk of *higher* memory fragmentation if we start to use
pymalloc for PyMem_Malloc()? Does someone know how to test it?

Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Git for Mercurial Users

[Nikolaus Rath]

Hello,

With the upcoming move to Git, I thought people might be 
interested in some thoughts that I wrote down when learning Git 
for the first time as a long-time Mercurial user:


http://www.rath.org/mercurial-for-git-users-and-vice-versa.html

Comments are welcome (but probably more appropriate off-list).


Best,
-Nikolaus 


--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Steve Dower]

On 03Feb2016 1120, Alexander Walters wrote:

Uh its C:\Anaconda[2]\ for anyone running the installer with the
privileges to edit the registry... (It wont ask to elevate unless you
install for all users, and that's where all users will install).  So on
that point alone, this saves nothing substantive really.  (I will go off
on python35 installing in insane locations some other time.)


The install location is customisable, and users can always write to 
their own registry hive.


The same applies to Python, so you can choose to install it as 
conveniently or as securely as you like. In either case, other 
applications need a guaranteed place to find these installations, and 
that place is the system registry.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Steve Dower]

On 03Feb2016 1851, eryk sun wrote:

On Wed, Feb 3, 2016 at 7:33 PM, Eric Snow  wrote:

Just wanted to quickly point out another use of the WIndows registry
in Python: WindowsRegistryFinder [1].  This is an import "meta-path"
finder that locates modules declared (*not* defined) in the registry.
I'm not familiar with the Windows registry nor do I know if anyone is
using this finder.


The "Modules" key (WindowsRegistryFinder in 3.3+ and previously
PyWin_FindRegisteredModule) adds individual modules by subkey name,
with the filepath in the default value (the filename can differ, but
it can't use an arbitrary extension). The "PythonPath" and "Modules"
keys both date back to Mark Hammond's Windows port in the mid 1990s.


Yep, essentially, I expect these keys that actually affect how Python 
works to remain under PythonCore, and continue not to be documented or 
recommended for general use. But I see no reason to deprecate or remove 
them.


Specialised situations that use these keys should continue to set them 
under PythonCore. I hope that is sufficiently implied by saying nothing 
about them in the PEP - I really don't want to have to be more explicit 
about it and I definitely do not want to actually name or list them in 
any way.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[eryk sun]

On Wed, Feb 3, 2016 at 7:33 PM, Eric Snow  wrote:
> Just wanted to quickly point out another use of the WIndows registry
> in Python: WindowsRegistryFinder [1].  This is an import "meta-path"
> finder that locates modules declared (*not* defined) in the registry.
> I'm not familiar with the Windows registry nor do I know if anyone is
> using this finder.

The "Modules" key (WindowsRegistryFinder in 3.3+ and previously
PyWin_FindRegisteredModule) adds individual modules by subkey name,
with the filepath in the default value (the filename can differ, but
it can't use an arbitrary extension). The "PythonPath" and "Modules"
keys both date back to Mark Hammond's Windows port in the mid 1990s.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Eric Snow]

On Tue, Feb 2, 2016 at 10:15 PM, Steve Dower  wrote:
> I was throwing around some ideas with colleagues about how we detect Python
> installations on Windows from within Visual Studio, and it came up that
> there are many Python distros that install into different locations but
> write the same registry entries. (I knew about this, of course, but this
> time I decided to do something.)
>
> [snip]
>
> So here is a rough proposal to standardise the registry keys that can be set
> on Windows in a way that (a) lets other installers besides the official ones
> have equal footing, (b) provides consistent search and resolution semantics
> for tools, and (c) includes slightly more rich metadata (such as display
> names and URLs). Presented in PEP-like form here, but if feedback suggests
> just putting it in the docs I'm okay with that too. It is fully backwards
> compatible with official releases of Python (at least back to 2.5, possibly
> further) and does not require modifications to Python or the official
> installer - it is purely codifying a superset of what we already do.
>
> Any and all feedback welcomed, especially from the owners of other distros,
> Python implementations or tools on the list.

Just wanted to quickly point out another use of the WIndows registry
in Python: WindowsRegistryFinder [1].  This is an import "meta-path"
finder that locates modules declared (*not* defined) in the registry.
I'm not familiar with the Windows registry nor do I know if anyone is
using this finder.  That said, ISTM the finder's use of the registry
does not face quite the same challenges you've described in the
proposal.  I expect Martin von Löwis could explain more as he was
involved with adding the finder.

Just wanted to throw that out there, particularly if there's a chance
of the finder's registry keys conflicting in some way.

-eric

[1] 
https://hg.python.org/cpython/file/5873cfb42ebe/Lib/importlib/_bootstrap_external.py#l570
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Steve Dower]

On 03Feb2016 0015, Alexander Walters wrote:

I am not saying this proposal will make the registry situation worse,
but it may break my solution to the headaches Python's registry use
causes with some non-standard module installers (and even the standard
distutils exe installers, but that is being mitigated).  In the wild
exist modules with their own EXE or MSI installers that check the
registry for 'the system python'.  No matter how hard you hit them, they
will only install to *that one python*.  You can imagine the sadist that
builds such an installer would not be receptive to the concept of
wheels.


(I agree, but maybe if you can point them to a PEP it'll help? Right 
now, there's nowhere to even point them to.)



So in order to force those modules to install to the python YOU
want, dammit, you have to edit the registry.  I have (contributed to) a
script that just sets whatever python it was last run with to the one
true system python.  Works with virtualenvs too.This is not a terribly
obscure script either, by the way.  It is in the first reply to the
google search for "how to install exes in a virtualenv".


I highly doubt it will break your current solution, as everyone pre- and 
post-update will still look in PythonCore\X.Y for Python.



So here I am in a situation where some pythons use another registry
entry.  I have no idea if this makes my life as a user harder. What does
this kind of meddling do against tools trying to autodetect python, and
finding an ever changing value?  Are we trying to guarantee that the
keys used by the python.org installers or only ever actually used by
standard CPython?


Guaranteeing that only python.org Python uses PythonCore is part of it, 
but the other part is officially telling everyone else that they are 
welcome to create their own keys.



I know for PTVS manually adding a python environment to visual studio is
trivial - you fill in three locations, and its done.  Just today I added
a python environment to my system that was not autodetected.  It took
under a minute and almost no effort to add it... so for that tool this
adds very little benefit.  I do not know about other tools.


I'm also a PTVS maintainer, so I know how much magic is going on behind 
those three locations :) But I don't think people should need to do that 
by hand at all.


For example, the path to an Anaconda installation is buried deep inside 
AppData (as is Python 3.5+ now), and varies based on your username. 
Canopy does the same, and once you've found it there are (or were?) at 
least three copies of python.exe to choose from (we worked with 
Enthought to make this Just Work for PTVS users).



On a very personal note (like the rest of this wasn't my personal issues
with possibly making my life slightly more difficult), I would much
rather see python stop touching the registry all together - but I have
no strong argument for that.


I also agree with that, but ultimately the registry is the global 
configuration store on Windows, and when we need global state it is the 
place to go. (My actual hope for Python 3.6 is to drop the few places 
where Python *reads* from the registry for configuration that shouldn't 
be global, at which point the Python key is solely for other programs. 
But those fixes are probably not PEP-worthy.)


Cheers,
Steve


___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Git for Mercurial Users

[Chris Angelico]

On Thu, Feb 4, 2016 at 10:38 AM, Nikolaus Rath  wrote:
> Hello,
>
> With the upcoming move to Git, I thought people might be interested in some
> thoughts that I wrote down when learning Git for the first time as a
> long-time Mercurial user:
>
> http://www.rath.org/mercurial-for-git-users-and-vice-versa.html
>
> Comments are welcome (but probably more appropriate off-list).

Also worth reading is this quick summary of roughly-equivalent commands:

https://github.com/sympy/sympy/wiki/Git-hg-rosetta-stone

ChrisA
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] speed.python.org

[Zachary Ware]

I'm happy to announce that speed.python.org is finally functional!
There's not much there yet, as each benchmark builder has only sent
one result so far (and one of those involved a bit of cheating on my
part), but it's there.

There are likely to be rough edges that still need smoothing out.
When you find them, please report them at
https://github.com/zware/codespeed/issues or on the sp...@python.org
mailing list.

Many thanks to Intel for funding the work to get it set up and to
Brett Cannon and Benjamin Peterson for their reviews.

Happy benchmarking,
-- 
Zach
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Steve Dower]

On 03Feb2016 0923, eryk sun wrote:

On Wed, Feb 3, 2016 at 10:46 AM, Steve Dower  wrote:


sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))


It seems like a bug (in spirit at least) that this step isn't skipped
for -E and -I (Py_IgnoreEnvironmentFlag, Py_IsolatedFlag).


They should be skipped. If not, I'm very much in favour of fixing that 
immediately in all active branches.



I haven't looked into pywin32's use of this recently - I tend to only use
Christoph Gohlke's  wheels that don't register anything.


I install the pypiwin32 wheel using pip, which uses pypiwin32.pth:

 # .pth file for the PyWin32 extensions
 win32
 win32\lib
 Pythonwin

 import 
os;os.environ["PATH"]+=(';'+os.path.join(sitedir,"pypiwin32_system32"))

This is different from a PythonPath subkey in a couple of respects.
The paths listed in .pth files are appended to sys.path instead of
prepended. They also don't get added when run with -S or for a venv
environment that excludes site-packages.


Yeah, there are serious problems with doing these kinds of hacks in .pth 
files. However, this is not directly affected by the registry, so 
thankfully not a concern right now.


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Steve Dower]

On 03Feb2016 0839, Paul Moore wrote:

On 3 February 2016 at 16:29, Steve Dower  wrote:

Final point I want to reiterate - Python itself is essentially registry free
already in that it does not need registry settings to function.


That's something we should probably publicise better. People seem
unaware of it (in much the same way that they never really noticed zip
application support). Maybe we could include a section in the Python
3.6 "What's new" (even though it's not technically new - but I did a
quick check of What's New back to 3.2 and couldn't see any mention)?


Maybe, but since it is still potentially problematic I'd rather not 
right now.


Basically, I don't want to have to support people whose "portable" 
version of Python works fine on one machine, but has syntax errors in 
the stdlib on another machine.


Adding the applocal option for 3.5 (described at 
https://docs.python.org/3.5/using/windows.html) helps with this, but I'm 
guessing always running in isolated mode is not what most people really 
want.


Until we genuinely never rely on the registry, I don't want to claim 
that we don't rely on the registry.


Cheers,
Steve

---

Rest of the email is spelling out how to create the scenario above, 
since I assume people won't believe it :)


1. Take Python 3.4.1, install it (Just for Me), zip up the stdlib into 
python34.zip and copy the binaries and zip to a "portable" folder


2. Update to Python 3.4.2 on the main machine

3. Run "-m test test_grammar" with your portable 3.4.1.

4. Boom! SyntaxError in test_grammar.py because you picked up the 3.4.2 
stdlib (the error comes from https://hg.python.org/cpython/rev/4ad33d82193d)



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Paul Moore]

On 3 February 2016 at 17:04, Steve Dower  wrote:
> Rest of the email is spelling out how to create the scenario above, since I
> assume people won't believe it :)
>
> 1. Take Python 3.4.1, install it (Just for Me), zip up the stdlib into
> python34.zip and copy the binaries and zip to a "portable" folder
>
> 2. Update to Python 3.4.2 on the main machine
>
> 3. Run "-m test test_grammar" with your portable 3.4.1.
>
> 4. Boom! SyntaxError in test_grammar.py because you picked up the 3.4.2
> stdlib (the error comes from https://hg.python.org/cpython/rev/4ad33d82193d)

Sigh. There's nothing so small that it isn't a compatibility break :-)

But of course this process violates the rule "set PYTHONHOME or have a
Lib/os.py *file* alongside python.exe". Like you say the rules are
subtle enough that people will make mistakes :-(

Thanks for the explanation.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[eryk sun]

On Wed, Feb 3, 2016 at 10:46 AM, Steve Dower  wrote:
>
> sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
> sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))

It seems like a bug (in spirit at least) that this step isn't skipped
for -E and -I (Py_IgnoreEnvironmentFlag, Py_IsolatedFlag).

> I haven't looked into pywin32's use of this recently - I tend to only use
> Christoph Gohlke's  wheels that don't register anything.

I install the pypiwin32 wheel using pip, which uses pypiwin32.pth:

# .pth file for the PyWin32 extensions
win32
win32\lib
Pythonwin

import 
os;os.environ["PATH"]+=(';'+os.path.join(sitedir,"pypiwin32_system32"))

This is different from a PythonPath subkey in a couple of respects.
The paths listed in .pth files are appended to sys.path instead of
prepended. They also don't get added when run with -S or for a venv
environment that excludes site-packages.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Paul Moore]

On 3 February 2016 at 16:46, Steve Dower  wrote:
> So a few high-level observations:
>
> * any program can install anywhere on the machine and make its libraries 
> available to a specific version of Python by creating a subkey under 
> 'PythonCore\x.y\PythonPath'

Yeah, that's horrid but not really something we can change without
compatibility breakage. But it's essentially opt-in - if you don't
need the feature you don't have to register anything under that key.

> * any environment lacking 'Lib\os.py' (e.g. venv) relies on the registry to 
> locate enough stdlib to import site
> * this is too complicated, but guaranteed we will break users in production 
> if we change it now
>
> So if repackagers follow a few rules (that I documented in 
> https://docs.python.org/3.5/using/windows.html - I see the process above is 
> also documented there, which I wish I remembered before writing all that 
> out), they'll be fine. Unfortunately, following those rules means that you 
> don't register anywhere that separate tools can find you, and so users 
> complain and you "fix" it by doing the wrong thing.

Thanks for the explanation. And for documenting it (even if I looked
for the documentation, failed to find it and then whined about it not
being documented - my apologies!)

More specifically for the people wanting "portable" Python systems, if
I read that right then if python.exe is alongside a Lib directory
containing os.py, Python needs no environment variables, and no
registry entries, to run perfectly. I don't see any issue with Python
builds that don't register themselves not showing up in tools that
look for Python installations. And as you say, if we give people who
make "official" distributions a way to properly register, then that
helps them and leaves the unregistered case for "homebrew" portable
copies of Python.

(I've just seen your other note about it being "potentially
problematic". OK, let's leave it low-key for now, but when we are
comfortable with it, can we publicise it then? I get a definite
impression that quite a lot of people assume that "you can't have a
portable build of Python").

> This PEP offers a right way to fix it.

Thanks for the explanation, and I now agree that's what the PEP is doing.

So +0.5 from me for this PEP (Only 0.5, because I still have some
concerns that talking about registry entries in such detail gives the
impression that Python is tied to them more than it actually is. If
you can see a way of toning down the wording, then great, but better
to document the proposal accurately than to water it down because
people might get a mistaken impression).

>> It's possible that the reason the above two points have been missed is
>> because the proposal focuses purely on "informational" registry data.
>> But Python also modifies sys.path based on the registry entries - and
>> possibly has other behavioural changes as well. The pywin32 package,
>> in particular, makes use of this (it's a feature of pywin32 that I
>> disagree with and I wish it didn't do that, but it does, and it's a
>> very widely used package on Windows). So ignoring this aspect of
>> Python's behaviour is a big problem. (Also, what changes will pywin32
>> need to make to correctly support being installed into non-python.org
>> distributions when this proposal is implemented?)
>
> I haven't looked into pywin32's use of this recently - I tend to only use 
> Christoph Gohlke's  wheels that don't register anything. But it is certainly 
> a valid concern. Hopefully Mark Hammond is watching :)

Yeah, I've not checked if pywin32 still does this, it's a long time
since I really used it. Like you, I go for wheels only these days.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Paul Moore]

On 3 February 2016 at 19:20, Alexander Walters  wrote:
> Uh its C:\Anaconda[2]\ for anyone running the installer with the
> privileges to edit the registry... (It wont ask to elevate unless you
> install for all users, and that's where all users will install).  So on that
> point alone, this saves nothing substantive really.

Per-user installs go into HKCU, which doesn't require elevation, so
the proposal *does* help for per-user installs. And does the all-users
install really offer no option for users to choose their own install
location? Not even to switch to another drive (if, for example, C: is
protected)?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Alexander Walters]

On 2/3/2016 11:17, Steve Dower wrote:

I know for PTVS manually adding a python environment to visual studio is
trivial - you fill in three locations, and its done.  Just today I added
a python environment to my system that was not autodetected.  It took
under a minute and almost no effort to add it... so for that tool this
adds very little benefit.  I do not know about other tools.


I'm also a PTVS maintainer, so I know how much magic is going on 
behind those three locations :) But I don't think people should need 
to do that by hand at all.


For example, the path to an Anaconda installation is buried deep 
inside AppData (as is Python 3.5+ now), and varies based on your 
username. Canopy does the same, and once you've found it there are (or 
were?) at least three copies of python.exe to choose from (we worked 
with Enthought to make this Just Work for PTVS users).
Uh its C:\Anaconda[2]\ for anyone running the installer with the 
privileges to edit the registry... (It wont ask to elevate unless you 
install for all users, and that's where all users will install).  So on 
that point alone, this saves nothing substantive really.  (I will go off 
on python35 installing in insane locations some other time.)

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Paul Moore]

On 3 February 2016 at 16:29, Steve Dower  wrote:
> Final point I want to reiterate - Python itself is essentially registry free
> already in that it does not need registry settings to function.

That's something we should probably publicise better. People seem
unaware of it (in much the same way that they never really noticed zip
application support). Maybe we could include a section in the Python
3.6 "What's new" (even though it's not technically new - but I did a
quick check of What's New back to 3.2 and couldn't see any mention)?

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Steve Dower]

On 03Feb2016 0150, Glenn Linderman wrote:

Portable software exists, but often is 3rd party hacks of popular FOSS
rather than being directly supported by the FOSS development team.
Python falls into this category. Happily, I recently found WinPython
Zero, which hacks it (somehow) to work portably, and I've deployed it
successfully on Dropbox. I'd rather Python were portable on its own,
without hacks.

Portability requires not using the registry, so I agree with Alexander
there.


Python has been registry independent for a while if you set the 
PYTHONHOME environment variable (and independent but potentially 
unreliable even without this).


Most of the registry settings created on install are for supporting 
upgrades, repairs and uninstallation, none of which matter in your case. 
(Also many python-dev readers' cases, but there are a lot of users who 
get into trouble very quickly without this level of management.)



So here are my preferences for CPython:

1) It would be best CPython itself were fully portable. That wherever
you point the installer, it lives there, and if somehow you happen to
execute it from that directory, it would use its invocation path as the
basis for finding the rest of the installed pieces.


Agreed, and it already basically does this as mentioned above.


2) A script could be provided that sets the association for .py and the
corresponding ftype to point to the python that executed the script...
and which has a parameter to clear that back out, too. This would be to
allow users to set a particular python for convenient script execution,
since Windows does the association thing, instead of using #! lines.


You probably want the py.exe launcher here (though that relies on Python 
being registered...), as it handles shebangs - even /usr/bin/env style.


A script such as what you're asking for would be possible, but not 
something I want to be responsible for providing and maintaining.



3) A script could be provided (maybe the one Alexander referred to) that
sets the registry so that the "apparently one true System Python" points
to the python that executed the script... and which has a parameter to
clear that back out, too.  This would be for compatible transition to
the new registry-free Python versions for packages that have weird
installers (such as Alexander alluded to). But with the registry-free
Python available, those packages would hopefully see the future is
registry free, and avoid requiring registry data to do installs.
4) A script could be provided to add the python that executed the script
to the PATH, with an option to remove it.

5) A script could be provided to add the python that executed the script
to the launcher .ini file, with an option to remove it.

6) A script could be provided to add the python that executed it to the
Start menu, and/or Desktop icons, with an option to remove them.


4-6 are basically the definition of an installer, and other installers 
are also able to do this.



Maybe scripts 2-6 are all the same one, with different options (or
different invocation shortcuts for the clicky folk). Not everyone would
probably need all the scripts, but none of them would be particularly
large, such that they would be burdensome for others to ignore.

Such a collection of scripts would allow folks to achieve various levels
of integration with Windows "conveniences", without requiring it. The
portability would allow Python to be installed on Dropbox or a network
share, and used from there without requiring every team member to do all
the individual installs of the packages needed for a project.


Perhaps you really want a script to run the installer and then pip?

Final point I want to reiterate - Python itself is essentially registry 
free already in that it does not need registry settings to function. The 
current problems are:


1. other programs need to locate all available Pythons
2. there appears to only be one space to register your Python
3. this space is *sometimes* used by Python to locate itself and 
installed packages


I want to fix problem 2 via documentation, and then look at the much 
more difficult problem 3.


Cheers,
Steve

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Steve Dower]

On 03Feb2016 0321, Paul Moore wrote:
> Some issues with this proposal:
> 
> 1. I don't like the way it states that Python distributions "must" set
> keys. I'd rather that it were explicitly stated that a distribution
> which sets no registry keys is entirely valid (with the obvious
> implication that tools which scan the registry looking for installed
> Python distributions won't be able to see such an installation).

Good point, I never meant to imply that. If you don't want your Python 
install/env found then you don't have to register anything.

(Of course, when users come to us IDE developers and say "your tool can't find 
Python XYZ", we'll all just go to Python XYZ and say "your users need you to 
register when you install, see this PEP" :) )

> 2. It's not clear to me how alternative distributions will specify
> their registry keys (or the fact that they don't use them). The
> registry keys are built into the Python interpreter - this proposal
> seems to imply that distributions that simply repackage the python.org
> build will no longer be valid or supported, and instead anyone who
> wants to produce a custom Python distribution (even private in-house
> repackagings) will need to change the Python source and rebuild.
> That's a pretty major change, and if that *is* the intent, then as a
> minimum I'd say we need to provide compiler flags to let
> redistributors specify their Company/Tag values (or say that they want
> to omit registry use). And I'm still not happy that "repackage the
> python.org binaries" has been removed as an option.

There's only one place where the registry key is used within the interpreter 
itself, which is PC/getpathp.c. Essentially the process is this:

sys.path = []
sys.path.append('')
sys.path.extend(os.getenv('PYTHONPATH').split(';'))
sys.path.extend(read_subkeys(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))
sys.path.extend(read_subkeys(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath\**'))

home = os.getenv('PYTHONHOME')
if not home:
if os.path.exists(os.path.join(sys.argv[0], '..', 'Lib', 'os.py')):
home = os.path.dirname(sys.argv[0])
if not home:
paths = 
read_value(fr'HKCU\Software\Python\PythonCore\{sys.winver}\PythonPath')
if not paths:
paths = 
read_value(fr'HKLM\Software\Python\PythonCore\{sys.winver}\PythonPath')
if paths:
sys.path.extend(paths.split(';'))
else:
sys.path.append(r'.\Lib')
# more well-known subdirs
else:
sys.path.append(os.path.join(home, 'Lib'))
# more well-known subdirs ...

So a few high-level observations:

* any program can install anywhere on the machine and make its libraries 
available to a specific version of Python by creating a subkey under 
'PythonCore\x.y\PythonPath'
* any environment lacking 'Lib\os.py' (e.g. venv) relies on the registry to 
locate enough stdlib to import site
* this is too complicated, but guaranteed we will break users in production if 
we change it now

So if repackagers follow a few rules (that I documented in 
https://docs.python.org/3.5/using/windows.html - I see the process above is 
also documented there, which I wish I remembered before writing all that out), 
they'll be fine. Unfortunately, following those rules means that you don't 
register anywhere that separate tools can find you, and so users complain and 
you "fix" it by doing the wrong thing.

This PEP offers a right way to fix it.

> It's possible that the reason the above two points have been missed is
> because the proposal focuses purely on "informational" registry data.
> But Python also modifies sys.path based on the registry entries - and
> possibly has other behavioural changes as well. The pywin32 package,
> in particular, makes use of this (it's a feature of pywin32 that I
> disagree with and I wish it didn't do that, but it does, and it's a
> very widely used package on Windows). So ignoring this aspect of
> Python's behaviour is a big problem. (Also, what changes will pywin32
> need to make to correctly support being installed into non-python.org
> distributions when this proposal is implemented?)

I haven't looked into pywin32's use of this recently - I tend to only use 
Christoph Gohlke's  wheels that don't register anything. But it is certainly a 
valid concern. Hopefully Mark Hammond is watching :)

Cheers,
Steve

> 
> Paul

___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Python environment registration in the Windows Registry

[Matthew Einhorn]

On Wed, Feb 3, 2016 at 3:15 AM, Alexander Walters 
wrote:

> ...just when I thought I have solved the registry headaches I have been
> dealing with...
>
> I am not saying this proposal will make the registry situation worse, but
> it may break my solution to the headaches Python's registry use causes with
> some non-standard module installers (and even the standard distutils exe
> installers, but that is being mitigated).  In the wild exist modules with
> their own EXE or MSI installers that check the registry for 'the system
> python'.  No matter how hard you hit them, they will only install to *that
> one python*.


If I remember correctly, you can use `wheel convert filename.exe` on those
installers which create a wheel that you can install. I think that's what I
used to do with pywin32 before pypiwin32 came along.

I just tested it and it still works on the pywin32 exe.
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com