[Distutils] Re: Question from a Beginner

[Steve Dower]

The main thing for you to do is to double-check all the names you type 
in *before* you install anything. Most of the "security" issues come 
down to people trying to catch misspellings ("typo-squatting"), so if 
you've spelled everything correctly, you'll get the packages you expected.


If you don't even trust *those* packages, or their dependencies, you're 
signing up for a whole lot more work (reviewing code, manually creating 
a private mirror, curation, etc.). Ultimately it will be up to you to 
decide who you trust and how much you trust them.


I believe the infrastructure itself to be trustworthy, and most of the 
people publishing popular packages are trustworthy. But ultimately 
you're on your own right now for detecting impersonation.


Cheers,
Steve

On 9/17/2021 5:13 PM, Sonic Emitter3000 wrote:
Hello, hope you're doing well. I greatly appreciate the effort of you 
people to make open source projects like you do, but I must ask.


I have heard that security is quite lax when installing modules using 
the most popular sites for Python modules. Would you know of how I would 
protect myself more from potentially malicious fakes of popular Python 
modules?



--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/MXFS2XS2QIT2NO2YIGAKYE43JQLB3PQV/

[Distutils] FYI: PEP 632: Deprecate distutils module

[Steve Dower]

Hi distutils-sig

Just wanted to let you know that I've proposed a core CPython PEP to 
formally deprecate and remove the standard library distutils module.


Text and discussion at 
https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134


The PEP has gone to python-dev formally because it only deals with the 
standard library module. setuptools, pip, and recent package work 
feature in the motivation, but are not bound to follow anything in this PEP.


Cheers,
Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/SSDVWVZBZRJLJG4SGW5OFBF3ZWPFPKZY/

[Distutils] Re: Twine 3.2.0 released

[Steve Dower]

Congrats! And thanks for stepping up to maintain one of our important tools!

On 24Jun2020 1114, Brian Rutledge wrote:

https://pypi.org/project/twine/3.2.0/

Changelog: https://twine.readthedocs.io/en/latest/changelog.html

This is my first release as a Twine maintainer. Thanks to:

- Sumana Harihareswara for getting me started, and nudging me along the way
- Ian Stapleton Cordasco, Jason R. Coombs, and Dustin Ingram for code 
review and guidance in general
- Devesh Kumar Singh, Felipe Mulinari Rocha Campos, and Vikram Jayanthi 
for substantial contributions to this release

--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/FYPVPW7UFQ2MZRHURLXCZRC2HOAG3726/

[Distutils] Re: package management - common storage while keeping the versions straight

[Steve Dower]

On 24Jun2020 1923, David Mathog wrote:

I think I will experiment a little with pipenv and if necessary after
each package install use a script to remove the installed libraries
and replace them with a hard link to the one in the common area.
Maybe it will be possible to put in those links before installing the
package of interest (like for scanpy, see first post), which will
hopefully keep it from having to rebuild all those packages too.


Here's a recent discussion about this exact idea (with a link to an 
earlier discussion on this list): 
https://discuss.python.org/t/proposal-sharing-distrbution-installations-in-general/2524


It's totally possible, though it's always a balance of trade-offs. Some 
of the people on that post may be interested in developing a tool to 
automate parts of the process.


Cheers,
Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/IUGVEOU5U7VYJMEUC6E3VWBE6OFTLPHR/

[Distutils] Who is approving PyPI size limit exceptions these days?

[Steve Dower]

These issues seem to be awaiting approval: 
https://github.com/pypa/pypi-support/labels/limit%20request


Most pressing (for me ;) ) is my colleagues at 
https://github.com/pypa/pypi-support/issues/457, who are already dealing 
with confused users not finding the packages they expect.


Thanks,
Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/5XGY5K2QZ5SJI5JU7GRMTWJ2FPDO3OV5/

[Distutils] Re: pySerial 2.7 - MSI

[Steve Dower]

On 17Sep.2019 0255, Saqib Rashid (UK) wrote:
> We are testing team working on Smart Metering Implementation Programme
> and use pySerial 2.7. Our IT team is preparing an OS build with all the
> required tools in it. To silently install the tool they them in MSI. I
> was wondering if you have this in MSI format and can share it with us?

Hi Saqib

Unless the maintainers of PySerial are hear and speak up, you will need
to reach them (I assume at https://github.com/pyserial/pyserial though I
have not checked).

However, as most Python projects are not distributed as MSI files, if
you require that then you will be best to create your own MSI package
with the files you need - in general, there is no registration required
and copying the files is sufficient. The tutorial at
https://www.firegiant.com/wix/tutorial/ will help you create an MSI.

Cheers,
Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/FLTPVX4NQSC2HACTDJ5XIEYNCU2RRY4Y/

[Distutils] Re: Python 3.7.4 MSI

[Steve Dower]

While this is technically true, your chances of getting it to work are 
very slim.


If you have a need for an MSI, my best suggestion right now is to take 
one of our distributions (the package at 
https://www.nuget.org/packages/python is likely easiest) and use a tool 
to generate one from that. We require additional maintainers if we are 
going to maintain more distribution mechanisms.


If your organisation is running Windows 10, you can use SCCM to deploy 
the Store package as well. Depending on your precise needs, that may be 
an option too.


Cheers,
Steve

On 09Sep2019 1716, Tzu-ping Chung wrote:
FWIW, individual msi files are still available on python.org, e.g. (for 
3.7.4 64-bit)


https://www.python.org/ftp/python/3.7.4/amd64/

This is essentially how the web-based installer works. Those msi files 
map quite nicely to individual options of the exe wrapper, so it 
shouldn’t be too difficult to figure out what you need (or you can just 
read the CPython source).


I assume existing versions are guaranteed to work as long as python.org 
works (otherwise the web-based installer would cease to work), but 
there’s likely no compatibility guarantees for future releases.


*From: *Paul Moore 
*Sent: *09 September 2019 22:29
*To: *Renegad3 Kay 
*Cc: *Distutils 
*Subject: *[Distutils] Re: Python 3.7.4 MSI

Correct. The installer technology used for the python.org builds

changed some time ago (I think Python 3.4 was the last version to use

an MSI installer).

Paul

On Mon, 9 Sep 2019 at 15:25, Renegad3 Kay  wrote:

 >

 > Greetings!

 >

 > my organization uses Python across it's departments but the recent 
versions of Python do NOT have an MSI download. We use SCCM for 
deployment of software and because the downloads are all .exe based, the 
program is no longer in compliance with my organization's security 
policies. I've looked and I've looked but I cannot find an MSI for this 
program.


 >

 > IT Guy.

--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/WFJXEITIDQ4D4MH3SBTPQDDWU3QEZEII/

[Distutils] Re: API for Adding Contributors to a Package?

[Steve Dower]

On 07May2019 1048, Philip James wrote:
Hi there! I took a look at the API docs that I could find 
(https://warehouse.readthedocs.io/api-reference/) and I couldn't see if 
it was possible to programmatically edit the contributors who have 
access to a project on PyPI.


My use case is for the BeeWare project being able to dynamically add and 
remove permissions to upload new versions of packages.


Does this API exist?

I'm also at PyCon until Thursday if someone wants to talk about my use 
case directly and is also here.


Is this significantly different from having a way to generate single-use 
or time-bounded API keys for uploading?


I've been interested in something like that for use from Azure Pipelines 
to make it easier to publish packages as part of builds.


Cheers,
Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/BYFPQDS6E4GLHZVFTCK2PWLPQNMEW5Q7/

[Distutils] Re: PEP-582 concerns

[Steve Dower]

On 20Feb2019 0927, Nathaniel Smith wrote:

That said, I prefer the approach of pipx
(https://pypi.org/project/pipx/) for scripts anyway. It too has the
problem of not updating your PATH for you, but at least it keeps tools
separate from dependencies, as they should be.


I think this is the third time we've had this conversation in a week
:-(. Pipx is great for the cases it targets, and if it's sufficient
for all of your use cases then that's great, but it isn't sufficient
for mine, and that's not because your use cases are right and mine are
wrong.


This should just be the description of distutils-sig :)

Cheers,
Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/5J3TPDCBTWFZ2ZF4S2OGU7R2WJFDOQVC/

[Distutils] Re: PEP-582 concerns

[Steve Dower]

On 20Feb2019 0831, Nathaniel Smith wrote:
Yeah, __pypackages__ has no way to handle scripts, and also no way to 
access packages when you're running from a directory. Pipenv already 
handles both of these cases fine today, so I'm not sure how having 
__pypackages__ several years from now could help you.


Uh, it totally has both. It has no way to handle updating your 
terminal's environment for you, but it can put scripts *somewhere* ;)


It can also handle accessing packages when running from your project 
directory. If you meant subdirectory, sure, that would be a major 
security issue to do that (much as I want to), but if you meant "both 
scripts and -m don't work" then that's just incorrect.


That said, I prefer the approach of pipx 
(https://pypi.org/project/pipx/) for scripts anyway. It too has the 
problem of not updating your PATH for you, but at least it keeps tools 
separate from dependencies, as they should be.


Cheers,
Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/5XYFHI556XMZLNOKPTOJJKFAE4F3AMUS/

[Distutils] Re: PEP-582 concerns

[Steve Dower]

On 20Feb2019 0839, Paul Moore wrote:

On Wed, 20 Feb 2019 at 16:28, Steve Dower  wrote:


To be totally clear, and maybe this needs to be in the PEP (probably in
three more various forms to make sure everyone gets it), you can emulate
most of the PEP today with "pip install --target __pypackages__/3.7 ..."
and "$env:PYTHONPATH = './__pypackages__/3.7'". Nothing else changes.
The advantage is that even this amount of friction goes away.


Sorry for the drive-by comment (I don't have time to read the PEP
right now) but does this mean that "pip install --upgrade" won't be
supported against __pypackages__ directories (it currently doesn't
work properly with --target, IIRC - certainly *some* things go weird
with --target) or will the PEP include mechanisms to allow pip to work
more cleanly with __pypackages__ than it currently does with --target?


The PEP explicitly doesn't say anything about what pip can/should do (at 
Donald's request), but the assumption is that it will support it 
properly. (Hence "*emulate* *most* of the PEP".)


My point is just that the intent is for it to be a "normal" package 
directory, not a reimagining of packaging.


Cheers,
Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/L7ZVTW34KAPDIRG2OEA2TIHSSMSRWD7Q/

[Distutils] Re: PEP-582 concerns

[Steve Dower]

On 20Feb2019 0803, Dan Ryan wrote:

One final thing this enables as far as I understand is a sort of npm-like 
option for ignoring resolution conflicts and simply performing a sort of nested 
installation of subdependencies inside a top level dependency’s __pypackages__ 
folder. So if you did install two packages with a conflict, they wouldn’t 
necessarily have to find a resolution.


I'm not sure where you got this idea from? It's certainly not part of 
the proposal.


To be totally clear, and maybe this needs to be in the PEP (probably in 
three more various forms to make sure everyone gets it), you can emulate 
most of the PEP today with "pip install --target __pypackages__/3.7 ..." 
and "$env:PYTHONPATH = './__pypackages__/3.7'". Nothing else changes. 
The advantage is that even this amount of friction goes away.



My only concerns with this PEP relate to bloat. I’d be interested in ways to 
reduce duplication across a system with many such folders.


Are you interested in ways to reduce duplication across a system with 
many virtual environments? Again, there's literally no difference here 
(except under the PEP there isn't a need to _also_ duplicate the Python 
binaries).


(And I snipped your concern about scripts/bin folders, but that's very 
much an point of open discussion. It really relies on some integration 
with whatever terminal a user happens to be using, and so far, the PEP 
does not concern itself with modifying PATH, only sys.path.)


Cheers,
Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/B4G7OZIWHTUOW7B2Z2DFI5YNXQPFMIQD/

[Distutils] Re: PEP-582 concerns

[Steve Dower]

On 20Feb.2019 0533, Tzu-ping Chung wrote:
> As one of the Pipenv maintainers, however, it is my personal opinion that this
> PEP would not be end up in the “yet another standard” situation, but even be
> beneficial to Pipenv, if done correctly.
> 
> I hope this can provide some confidence :)

I'd love to hear more about how Pipenv would make use of it. So far it's
only really been designed with pip in mind (and their team in the
discussion), but we've explicitly left it _very_ tool independent. So if
you can describe how it would work with Pipenv, that would be helpful
for finding things that need changing.

Also, the `pythonloc` package is an implementation of this that anyone
can try out today - https://pypi.org/project/pythonloc/ (the major
difference is that when implemented, you won't have to use "pythonloc"
and "piploc" to get the new behaviour).

Cheers,
Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/JBEBTV4DBFMQ6GUUJPREFKYTP5TUAKF4/

[Distutils] Re: PEP-582 concerns

[Steve Dower]

On 20Feb.2019 0556, Nathaniel Smith wrote:
> I wonder if we should stick a header on the PEP draft saying something
> like this? There's a lot of scattershot responses happening and I think
> a lot of the people reacting are lacking context.

I agree, I think given the amount of attention it's getting justifies
making more clear than usual that it has not even been considered for
acceptance. (Maybe we could even sneakily suggest that people could show
their support by helping pip/virtualenv/pipenv burn down their bug count
so they're ready to implement it ;) )

At the same time, it's nice to have a PEP be "controversial" for good
reasons for a change. Nobody really hates it, they mostly just misread
it or have different scopes in mind.

Cheers,
Steve
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/archives/list/distutils-sig@python.org/message/5VWO5CKLTAGLIZPX5ZTUE4O2I6NKOWQB/

[Distutils] Re: sudo pip install: install pip files into /usr/local on Linux?

[Steve Dower]

On 25May2018 0911, Victor Stinner wrote:

As an user, I want to use "sudo pip install" because packages
installed in /usr (or /usr/local) are accessible without having to
touch PYTHONPATH: the install directory is part of the default
sys.path.

Steve Dower also proposed the idea of a "default virtual environment"
somewhere in the $HOME directory which would be in the default
sys.path.


For anyone who wants to follow along with this proposal (which barely 
resembles the brief description here :) ), we've been working on it on 
Kushal's github repo: https://github.com/kushaldas/peps/pull/1/files


Clarifying comments/questions are welcomed. (Apologies to anyone who 
doesn't have a github account yet, but it seems like it will be getting 
increasingly harder to contribute to Python in any way without one, so 
you may not have much choice...)


Cheers,
Steve
--
Distutils-SIG mailing list
distutils-sig@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/7MCIV4CW4NYWPNYNHZX6S7N5OYWA2AUC/

[Distutils] Re: Python 3.7 binary wheels

[Steve Dower]

On 08May2018 2134, Nathaniel Smith wrote:
> for 3.6 there was a last minute problem
> with the Windows ABI that only got discovered during the rc period. But
> if you're willing to keep an ear out for that sort of thing, go wild.

I thought this was 3.5 (or maybe I've blanked out the more recent one)?
And the only reason we discovered it is because someone started building
wheels just before release. So please start building your wheels and let
us know as soon as possible if anything seems wrong!

Thanks,
Steve
--
Distutils-SIG mailing list
distutils-sig@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/SB2BKEMXQS2BD25XDUGUTATGXPTOWXSW/

Re: [Distutils] Should abi3 tag be usable on Windows?

[Steve Dower]

There is a solution to that problem on the linked issue. Basically, you need to 
declare Py_LIMITED_API in your code or as an extra preprocessor variable.

Windows doesn’t use a filename suffix for python3.dll linked extensions as it 
will be handled at load time. The tag for the wheel is outside of my area, so 
hopefully someone can chime in on that for you.

Cheers,
Steve

Top-posted from my Windows phone

From: Cosimo Lupo
Sent: Monday, January 8, 2018 10:32
To: distutils-sig@python.org
Subject: Re: [Distutils] Should abi3 tag be usable on Windows?

It turns out setuptools is still linking to PYTHON36.DLL instead of PYTHNO3.DLL 
even when py_limited_api=True
https://github.com/pypa/setuptools/issues/1248

This is a different, though somewhat related, problem from the one concering 
wheel/pip pep425tags disallowing `abi3` tag on Windows.


On Sun, Jan 7, 2018 at 12:39 PM Cosimo Lupo  wrote:
I jut found this related pip issue: https://github.com/pypa/pip/issues/4445

(I myself as @anthrotype commented on that thread back in April last year but 
then completely forgot...)

After re-reading it now, it's still not clear to me what the resolution on that 
issue was.

On Sun, Jan 7, 2018 at 10:12 AM Cosimo Lupo  wrote:
Hello,

CFFI has recently added support for Py_LIMITED_API extension modules built for 
CPython 3.
The wheel module since version 0.30.0 also supports passing —py-limited-api 
cp3X to bdist_wheel command to allow the generated .whl to be installed on all 
CPython versions equal or greater than the one specified.

Yesterday I was trying to apply this on a cffi-built extension module, and it 
worked for Linux and macOS but failed for Windows:

https://github.com/hynek/argon2_cffi/pull/32

The AssertionError from wheel.pep425tags complains that a tag with abi3 would 
be unsupported for the target platform.

Alex Gronholm commented

> imp.get_suffixes() does not seem to contain any ABI3 suffixes, but I'm not 
> sure if this is even applicable on Windows.

Incidentally, I noticed one specific package, PyQt5, that distributes both 
abi3-tagged wheels for Mac and manylinux and Windows wheels for a range of 
cp35.cp36.cp37 but with abi tag set as “none”, and they do seem to work.

So, can one make such py_limited_api wheels work on Windows with the current 
state of the tooling, and if so how?

Thank you in advance

Cosimo Lupo


-- 
Cosimo Lupo


-- 
Cosimo Lupo

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[Steve Dower]

It must, or it couldn’t have a build time dependency on distutils :)

Top-posted from my Windows phone

From: Donald Stufft
Sent: Saturday, September 30, 2017 18:15
To: xoviat
Cc: Steve Dower; DistUtils mailing list
Subject: Re: [Distutils] Extracting distutils into setuptools

I think that the CPython builds a python executable, then uses that built 
executable to finish the installation.


On Sep 30, 2017, at 9:11 PM, xoviat <xov...@gmail.com> wrote:

It would be nice to know whether this information is correct, or whether I hold 
an invalid belief.

2017-09-30 20:09 GMT-05:00 xoviat <xov...@gmail.com>:
I have personally not built Python myself (though I've built many an 
extension), but what I can say is that I got the idea from Larry Hastings. 
According to him (this if for the Gilectomy fork):

"Second, as you hack on the Gilectomy you may break your "python" executable 
rather badly. This is of course expected. However, the python Makefile itself 
depends on having a working local python interpreter, so when you break that 
you often break your build too."

2017-09-30 19:59 GMT-05:00 Donald Stufft <don...@stufft.io>:



On Sep 30, 2017, at 3:52 PM, xoviat <xov...@gmail.com> wrote:

I don't think CPython needs to bundle all of its build-time dependencies. That 
principle doesn't really apply to other Python programs nor most other programs 
in general. AFAIK, CPython already has a build-time dependency on another, 
external, Python, so it wouldn't be too much to require the external Python to 
have setuptools installed with something like pyproject.toml (other programming 
languages usually bootstrap themselves with previous versions of the language 
along with some associated build tools).

As far as I can tell, CPython does *not* have a build time dependency on having 
Python available. I just spun up a bare alpine linux container and compiled 
CPython `master` branch on it. As far as I can tell the only Python that exists 
in this container is the one I just compiled.

That means that in order for CPython to depend on distutils to build as you 
indicate, it would also need to start depending on an existing version of 
Python being available. I don’t think that’s a great idea. I think Python 
should not depend on Python to build.




___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[Steve Dower]

On 29Sep2017 1306, Donald Stufft wrote:
There are bits of distutils that are super useful (Steve indicated the 
compiler support as one of them), and I think a better path forward is 
to do like we’ve done with the packaging library. Make a compiler 
package that can handle those bits and recommend that projects that want 
to handle compilation just reuse them (of course not mandate it!). Then 
we can focus first on creating a good API in that package without 
inheriting the problems of distutils, and then we can maybe work on 
bundling that inside of distutils and turn the distutils API into a shim 
over that (or just leaving it alone).


I'm happy enough with this approach, my only problem with it is that I 
don't want to be maintaining two versions (the new one that we want 
people to change to, and the old one so that people keep working with 
new versions of Python without having to change to the new one).


If we don't make a very clear statement that "distutils is no longer 
being maintained, it probably won't build valid extensions on 
3.[7/8/whatever], and we mean it this time" then it doesn't count. I 
insist on at least removing the distutils tests from the test suite, 
since as long as those need to pass I can never stop maintaining the 
stdlib version. And that puts this in the realm of python-dev, which 
means a PEP to formally deprecate distutils (that I'm happy to 
co-author) and recommend an alternative, as well as the alternative path 
needed to build the core extension modules on non-Windows platforms.


Whatever breaks setuptools's dependency on distutils helps drastically 
with this, as then we can recommend setuptools as the alternative path. 
Perhaps we can extract distutils into an installable package that will 
shadow the stdlib one, and setuptools can continue monkey patching while 
also taking a build/install-time dependency on it.


I honestly don't know what the best approach here is, which is why I 
raise the question. I'm fairly convinced that the best approach is *not* 
to have a whole lot of PEP 517 build tools that depend on distutils for 
core functionality.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Steve Dower]

That totally works for me. It also avoids the argument we seem to be having 
around tricking backends into creating the correct output by controlling their 
input. It’s the backend job to get the output correct – pip can output as many 
messages as it likes to blame someone else, if the aim is to avoid bugs being 
reported in the wrong place, but at some point we need to trust the backend to 
get it right even in a dirty source directory.

Top-posted from my Windows phone at EuroPython

From: Daniel Holth
Sent: Sunday, July 16, 2017 21:18
To: Steve Dower; Nathaniel Smith; Nick Coghlan
Cc: distutils-sig
Subject: Re: [Distutils] A possible refactor/streamlining of PEP 517

I agree that sdist consistency is not enforceable. Very little is. What if we 
deleted every unenforceable part of the PEP? No explanations of what backends 
should do. Every parameter is a hint. If you put the output file where 
requested then you are a good back end. Would that work better?

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Steve Dower]

Just throwing in a vote, since I am following along – Nathaniel is totally 
correct and I have no idea what Nick is talking about :)

In/out-of-place builds can’t guarantee that any “build wheel” operation will be 
consistent with the “build sdist” operation, except for dealing with a very 
narrow set of bugs. If you want wheels and sdists to be the same, require it of 
backends through the specification – not the API.

About the only option available here in the protocol would be a “strict” flag, 
which might allow backends to fail immediately if they can’t guarantee it (e.g. 
the example backend), but there is literally no API design that can implicitly 
force the correct behavior.

“Build wheel and make a mess in my repo” vs “build wheel without leaving a 
mess” (in-tree vs. out of tree) is a useful option, but not a solution to 
backends producing incorrect output.

(In case it’s not clear, I’m using Nathaniel’s definitions, since I don’t 
understand any of the other definitions people have used.)

Cheers,
Steve

Top-posted from my Windows phone

From: Nathaniel Smith
Sent: Sunday, July 16, 2017 10:25
To: Nick Coghlan
Cc: distutils-sig
Subject: Re: [Distutils] A possible refactor/streamlining of PEP 517

On Sat, Jul 15, 2017 at 11:27 PM, Nick Coghlan  wrote:
> On 16 July 2017 at 14:56, Nathaniel Smith  wrote:
>> But... that is not what the in-place/out-of-place distinction means in
>> normal usage, it's not the distinction that any of those build systems
>> you were surveying implement, and it's not the distinction specified
>> in the current PEP text.
>>
>> If what we want is a distinction between "please give me a correct
>> wheel" and "please give me a wheel but I don't care if it's broken",
>> then wouldn't it make more sense to have a simple flag saying *that*?
>
> No, because pip *also* wants the ability to request that the backend
> put the intermediate build artifacts in a particular place,

Say what? Where did they say this? I'm 99% sure this is just not true.

> *and*
> having that ability will likely prove beneficial given directory based
> caching schemes in build automation pipelines (with BitBucket
> Pipelines and OpenShift Image Streams being the two I'm personally
> familiar with, but it's a logical enough approach to speeding up build
> pipelines that I'm sure there are others).

Yeah, this is a really neat idea! I'm genuinely enthusiastic about it.
Which... is why I think this is a great target for a future PEP, that
can adequately address the complications that the current PEP is
skimming over. As I argued in a previous email, I think these
pipelines would actually *prefer* that out-of-place build be an
optional feature, but it's basically impossible to even have the
discussion about what they want properly as part of the core PEP 517
discussion.

> It just turns out that we can piggy back off that in-place/out-of-tree
> distinction to *also* indicate how much the frontend cares about
> consistency with sdist builds (which the PEP previously *didn't* say,
> but explicit text along those lines was added as part of
> https://github.com/python/peps/pull/310/files based on this latest
> discussion).

Okay, but then this is... bad. You're taking two unrelated
distinctions (in-place/out-of-place and sloppy/precise) and smashing
them together. In particular, one of the types of build that Donald
has said that he considers "sloppy" and worries about avoiding is any
kind of incremental build. So if we take Donald's concern and your new
PEP text literally, then it rules out incremental out-of-place builds.
But incremental out-of-place builds are exactly what you need for the
build pipeline case that you're citing as a motivation for this
feature.

Your PEP is trying to do too many things at once, and that means it's
going to do them poorly.

>> And in what case would a frontend ever set this
>> give_me_a_correct_wheel flag to False?
>
> When the frontend either genuinely doesn't care (hopefully rare, but
> not inconceivable), or else when its building from an unpacked sdist
> and hence can be confident that the artifacts will be consistent with
> each other regardless of how the backend handles the situation
> (expected to be very common, since it's the path that will be followed
> for sdists published to PyPI, and when handed an arbitrary PEP 517
> source tree to build, pip will likely try "build_sdist -> in-place
> build_wheel" first and only fall back to "out-of-tree build_wheel" if
> the initial build_sdist call fails).

Ah, the unpacked sdist case is a good point, I neglected that in my
discussion of a possible setuptools build_wheel hook. But it's fine --
even if we say that the setuptools build_wheel hook has to produce an
"sdist-consistent" wheel in all cases, then it can detect whether it's
building from an unpacked sdist (e.g. by keying off the presence of
PKG-INFO), and in that case it knows MANIFEST.in has already been
taken into account and it 

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Steve Dower]

One nice thing about providing a “put your work in this directory” setting for 
all tasks is that only the front end has to know how and where to create it, 
and how and when to clean it up later. Users may want to configure this across 
all projects, regardless of the backend in use.

Permitting this directory to be the source tree implicitly requires backends to 
support “in place” builds (i.e. you should put output files in a matching 
structure under that directory in case it really is the source tree). In this 
case, front ends need to be responsible for (not) running rmtree and backends 
should not blindly delete everything (or else they’ll get bug reports from very 
upset users).

Cheers,
Steve

Top-posted from my Windows phone at EuroPython

From: Thomas Kluyver
Sent: Monday, July 10, 2017 9:14
To: distutils-sig@python.org
Subject: Re: [Distutils] A possible refactor/streamlining of PEP 517

On Mon, Jul 10, 2017, at 07:01 AM, Nick Coghlan wrote:
> So I think we have pretty solid evidence that the reason the
> procedural "build directory preparation" hook wasn't sitting well with
> people was because that isn't the way build systems typically model
> the concept, while a "build directory" setting is very common (even if
> that "setting" is "the current working directory when configuring or
> running the build").

Hooray! :-)

Do we want to also provide a build_directory for the build_sdist hook?
In principle, I don't think making an sdist should involve a build step,
but I know that some projects do perform steps like cython code gen or
JS minification before making the sdist. I think this was a workaround
to ease installation before wheel support was widespread, and I'd be
inclined to discourage it now, so my preference would be no
build_directory parameter for build_sdist. Backends which insist on
generating intermediates at that point can make a temp dir themselves.

Then I guess that the choice between building a wheel directly and
attempting to build an sdist first (with direct fallback) is one for
frontends, and doesn't need to be specified.

Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] A possible refactor/streamlining of PEP 517

[Steve Dower]

Big +1 for passing the option to the backend rather than making the frontend 
figure out how to do each backend’s job for them.

And add MSBuild to the list, which trivially supports separate source, 
intermediate and output directories, as well as incremental rebuilds. (And 
recently got full x-plat support, so could potentially be useful on all 
platforms.)

Cheers,
Steve

Top-posted from my Windows phone

From: Nick Coghlan
Sent: Saturday, July 8, 2017 7:59
To: Nathaniel Smith
Cc: distutils-sig
Subject: Re: [Distutils] A possible refactor/streamlining of PEP 517

On 8 July 2017 at 13:36, Nathaniel Smith  wrote:
> On Fri, Jul 7, 2017 at 6:23 AM, Daniel Holth  wrote:
>> FYI distutils supports out of tree builds too. It is the -b argument to
>> 'setup.py build'.
>
> In theory, yes, but in practice, there are lots of setup.py files out
> there that mutate the source directory. For example, every project
> using Cython.
>
> I don't think a distutils/setuptools backend could comply with
> Thomas's wording except by doing a copytree or sdist or something
> just-in-case.

Which is fine - the point of making it a parameter instead of a
separate hook is that the folks in the best position to know whether
or not a particular build system has native out-of-tree build support
are the developers of that particular PEP 517 backend.

Since we know Scons supports this natively (by way of the
"variant_dir" setting), I went and checked some of the other build
systems that folks may end up wanting to develop backends for.

Native build system support for out-of-tree builds, so developers of
these shims should be able to delegate handling "build_directory":

- Scons: set variant_dir appropriately
(http://scons.org/doc/production/HTML/scons-user.html#chap-separate)
- meson: *only* supports out-of-tree builds
(http://mesonbuild.com/Using-multiple-build-directories.html)
- waf: configure the output directory appropriately
(https://waf.io/book/#_custom_build_outputs)
- premake: set targetdir appropriately
(https://github.com/premake/premake-core/wiki/targetdir)
- CMake: simplest way seems to be to save the initial working
directory as the source directory, cd to the build directory, then run
"cmake "
- yotta; based on CMake, so same approach applies
- autotools/make: similar approach to CMake, but running
"/configure && make " from the build
directory
- maven: require that projects using the shim support a configurable
build dir 
(https://stackoverflow.com/questions/13173063/out-of-tree-build-with-maven-is-it-possible/13173140#13173140)
- ant: also uses pom.xml files, so same approach as maven applies
- cargo: set CARGO_TARGET_DIR
(http://doc.crates.io/environment-variables.html#environment-variables-cargo-reads)

Explicitly willing to add native out-of-tree wheel creation based on
PEP 517's requirements:

- flit

Mostly assume in-place builds, no generic support for out-of-tree
builds that I can find, so developers of these shims will need to work
out how to handle "build_directory" (probably by copying the relevant
input files into the specified build directory and then doing an
in-place build, similar to the way Scons handles "variant_dir" by
default):

- setuptools/distutils
- gyp/ninja (including node-gyp)
- gradle

So I think the folks saying "We don't think a separate hook for build
directory preparation is the right abstraction" have a strong point,
as that approach doesn't appear to map cleanly to the design of *any*
popular build system for precompiled languages.

By contrast, "build_directory" as an optional setting for the binary
build command *does* map cleanly as a concept in many cases, and for
the ones where it doesn't, I readily found questions from other folks
also wanting to know how to do it, and PEP 517 interface shims copying
the files they care about then doing an in-place build is a perfectly
acceptable technical solution.

And as an added bonus, if a backend starts out by blindly doing
"shutil.copytree(source_directory)" for out-of-tree builds, then the
potential performance implications of that approach will become more
clearly a discussion between the projects using the backend and the
developers of the backend, rather than being something that frontends
like pip will need to worry about handling in the general case.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Steve Dower]

On 25May2017 0756, Paul Moore wrote:

On 25 May 2017 at 15:38, Nick Coghlan  wrote:

So I'm inclined to accept the encoding amendment, and then
provisionally accept the overall PEP pending implementation in pip.


Me too. (Assuming I understand Steve's comments on backends, and he's
comfortable with the idea that backends need to capture and manage
MSVC output for presentation to the frontend).


Sounds like you understood my comments :) +1 overall (-0 on a formal way 
to pass logs via the disk)


As I mentioned at one point, there's a bug against the CPython test 
suite that the distutils tests show too much console output, which is 
because distutils currently just lets MSVC write directly to the 
console. To fix it, we need to capture the output and then conditionally 
display it, at which point transcoding from ANSI to UTF-8 with 'replace' 
is trivial, and saves the front end (in this case, the test suite) from 
having to guess. So it is something that the backend around MSVC needs 
to do regardless, and if the PEP says "send me UTF-8" then it's one less 
thing for the backend developer to guess.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Steve Dower]

On 22May2017 1253, Paul Moore wrote:

It seems to me there are 2 schools of thought:

1. There are likely to be fewer front ends than back ends, and so the
front end(s) (basically, pip) should deal with the problem. Also,
backends are more likely to be written by developers who are looking
at very specific scenarios, and asking them to handle all the
complexities of robust multilingual coding is raising the bar on
writing a backend too high.

2. The backend is where the problem lies, and so the backend should
address the issue. Furthermore, a well-established principle in
dealing with encodings is to convert to strings right at the boundary
of the application, and in this case the backend is the only code that
has access to that boundary.

(I tend towards (2), but I honestly can't say to what extent that's
because it makes it "someone else's problem" for me ;-))


I also tend towards 2, and I assume I am one of the more likely people 
to write the part that invokes Microsoft's cl.exe/link.exe :)


Is the front end going to be directly invoking those tools? I would 
assume not, otherwise it won't be cross platform.


Since the shim belongs to the front end, I've essentially been ignoring 
it. The shim can invoke another part of the build tool, but that is not 
going to be cl.exe/link.exe either.


At some point there will be a script that runs the tools directly. I 
have been referring to that as the backend, and it is the part that 
should handle capturing and transcoding the output. Everything from 
there can be utf8:replace to prevent crashing, but we can't say "the 
frontend can handle all encodings", and shouldn't say "the frontend will 
only use bad encodings".


IMHO, #2 is definitely the right way to go. Yes, the platform specific 
code now has to worry about the encoding, but... the encoding is 
platform specific? So... that seems exactly right? :) Maybe I'm still 
missing something here, but I'm totally happy to leave it to Thomas to 
decide (which I think he has, but I haven't gotten to looking at that PR 
yet).


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] python 3.6.1

[Steve Dower]

Hi Jost

For Windows, those commands are typically not available by default (on PATH) 
because it is difficult to automatically add them.

You should be able to use “py.exe -m pip” as a substitute for “pip”. Or you can 
modify your default PATH environment variable to include your Python directory 
and the Scripts directory inside of it.

VS Code is basically just an editor. It does not have functionality to help 
with packages (the full Visual Studio 2017 does if you select the Python option 
when installing).

Cheers,
Steve

Top-posted from my Windows phone

From: Jost now
Sent: Sunday, May 21, 2017 4:18
To: distutils-sig@python.org
Subject: [Distutils] python 3.6.1

My Python does not recognise pip or pip3. I should be installed in the 64 bit 
executable I think.

Did I forget something? (I first installed the 32 bit version by mistake but 
deinstalled it)

I also installed VScode, and in that the DON Jayamanne python editor. Is that 
only an editor?, It does  Diagnostics but does not seen to be able to execute 
python.

I should also install a driver to my INVIDIA I was told, but told I could do 
that later, not sure how.

Any advice please.

PS: I always have the same problem with downloading programs, thet does not 
seem to be any guides that actually Works?


Sendt fra Mail til Windows 10


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Steve Dower]

On 20May2017 1315, Paul Moore wrote:

On 20 May 2017 at 17:36, Steve Dower <steve.do...@python.org> wrote:

In general, since most subprocesses (at least on Windows) do not have
customizable encodings, the tool that launches them needs to know what the
encoding is. Since we don't live in a Python 3.6 world quite yet, that means
the tool should read raw bytes from the compiler and encode them to UTF-8.


Did you spot my point that Visual C produces output that's a mixture
of OEM and ANSI codepages?


[SNIP]

Yes, and it's a perfect example of why the MSVC-specific wrapper should 
be the one to deal with tool encodings. If you forward unencoded bytes 
like this back to the UI, it will have to deal with the mixed encoding.



I'd be very surprised if build tool developers got this sort of edge
case correct without at least some guidance from the PEP on the sorts
of things they need to consider. You suggest "read raw bytes and
encode them to UTF-8" - but you don't encode bytes, you encode
strings, so you still need to convert those bytes to a string first,
and there's no encoding you can reliably use for this. You need to use
"errors=replace" to ensure you can handle inconsistently encoded data
without getting an exception.


The "read raw bytes and [transcode] them" comment was meant to be that 
sort of help. I didn't go as far as writing 
`output.decode(output_encoding, errors="replace").encode("utf-8", 
errors="replace")`, but that's basically what I meant to imply. The 
build tool developer is the *only* developer who can get this right, and 
if they can't, then they have to figure out the most appropriate way to 
work around the fact that they can't.


As for defining distutils as incompatible with the PEP, I'm okay with 
that. Updating distutils to use subprocess for launching tools rather 
than spawnv would be a very good start (and likely a good contribution 
for a new contributor), but allowing build tools to continue to be 
written badly is not worthwhile.


Cheers,
Steve

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Steve Dower]

On 20May2017 1011, Thomas Kluyver wrote:

On Sat, May 20, 2017, at 05:36 PM, Steve Dower wrote:

In general, since most subprocesses (at least on Windows) do not have
customizable encodings, the tool that launches them needs to know what
the encoding is. Since we don't live in a Python 3.6 world quite yet,
that means the tool should read raw bytes from the compiler and encode
them to UTF-8.


I half agree, but:
- Build tools may not 100% know what encoding output will be produced,
especially if the developer can supply a custom command for the build
tool to run.


In this case, the whole thing breaks down anyway. UI can't be expected 
to reliably display text from an unknown encoding - at some point it has 
to be forced into a known quantity, and IMHO the code closest to the 
tool should do it.



- It's possible for data on a pipe to be binary data with no meaning as
text.


Sure, but it cannot be rendered unless you choose an encoding. All you 
can do is dump it to a file (and let a file editor choose an encoding).



- As a lazy developer, I don't want to read stdout/stderr from a
subprocess only to spit it back to my own stdout/stderr. I'd much rather
just launch the subprocess and let it use the same stdout/stderr as my
build tool.


One of the open issues against distutils is that it does this. We can 
allow it, but a well-defined tool should capture the output and pass it 
to the UI component rather than bypassing the UI component.



So I think it's most practical to recommend that build tools produce
UTF-8 (if not sys.stdout.isatty()), but let build tool developers decide
how far they go to comply with that.


Require that build tools either send UTF-8 to the UI component, or write 
bytes to a file and call it a build output. I see no benefit in 
requiring both the build tool and the UI tool to guess what the text 
encoding is.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 - specifying build system in pyproject.toml

[Steve Dower]

On 20May2017 0820, Nick Coghlan wrote:

Good point regarding the fact that the Windows 16-bit APIs only come
into play for interactive sessions (even in 3.6+), while for PEP 517
we're specifically interested in the 8-bit pipes used to communicate
with build subprocesses launched by an installation tool.


I need to catch up on the PEP (and thanks Brett for alerting me to the 
thread), but this comment in particular cements the mental diagram I 
have right now:


(build UI) <--> (build tool) <--> (compiler)
( Python ) <--> (  Python  ) <--> (anything)

I'll probably read the PEP closely and see that this is entirely 
incorrect, but if it's right:


* encoding for text between the build UI and build tool should just be 
specified once for all platforms (i.e. use UTF-8).
* encoding for text between build tool and the compiler depends on the 
compiler


In general, since most subprocesses (at least on Windows) do not have 
customizable encodings, the tool that launches them needs to know what 
the encoding is. Since we don't live in a Python 3.6 world quite yet, 
that means the tool should read raw bytes from the compiler and encode 
them to UTF-8.


The encoding between the tool and UI is essentially irrelevant - the UI 
is going to transform the data anyway for display, and the tool is going 
to have to transform it from the compilation tools, so the best we can 
do is pick the most likely encoding to avoid too many operations. UTF-8 
is probably that.


That's my 0.02AUD based on a vague memory of the PEP and this thread. As 
I get time today (at PyCon) to read up on it I may post amendments, but 
in general I'm +100 on "just pick an encoding and make the 
implementations transcode".


Cheers,
Steve

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] GnuPG signatures on PyPI: why so few?

[Steve Dower]

Another drive-by contribution: what if twine printed the hashes for anything it 
uploads with a message basically saying "here are the things you should publish 
somewhere for this release so people can check the validity of your packages 
after they download them"?

I suspect many publishers have never considered this is something they could or 
should do. Some very basic prompting could easily lead to it becoming part of 
the normal workflow.

Top-posted from my Windows Phone

-Original Message-
From: "Nick Coghlan" 
Sent: ‎3/‎13/‎2017 0:53
To: "Glyph Lefkowitz" 
Cc: "DistUtils mailing list" ; "Ben Finney" 

Subject: Re: [Distutils] GnuPG signatures on PyPI: why so few?

On 13 March 2017 at 05:51, Glyph Lefkowitz  wrote:

To summarize: Even if we only cared about supplying package upstreams to Debian 
(and that is a tiny part of PyPI's mission), right now, using the existing 
tooling of uscan and lintian, the only security value that could _possibly_ be 
conveyed here would be an out-of-band conversation between the maintainer and 
upstream about what their signing keys are and how the signing process works.  
Any kind of automation would make it less likely that would happen, which means 
that providing tool support to automate this process would actually make things 
worse.


And much of the same benefits can be obtained by Debian and other third parties 
maintaining "known hashes" for historical PyPI releases and complaining if they 
ever change.


The only aspect that end-to-end package signing can potentially help with is 
bypassing PyPI as a potential point of compromise for *new* never-before-seen 
releases, and much of *that* benefit can be gained by way of publishers 
providing a list of "expected artifact hashes" through a trusted channel that 
they control and the PyPI service can't influence.


GPG signatures of the artifacts themselves is just one way of establishing that 
trusted information channel, and it's a particularly publisher-hostile one 
that's also pretty end-user-hostile as well.


The TUF based approach in PEP 458 and PEP 480 has at least in principle support 
from both Donald and I, but in addition to still relying on HTTPS to bootstrap 
initial trust, it is also gated behind the Warehouse migration and shutting 
down the legacy PyPI implementation (which is a sufficiently tedious activity 
that we think the chances of achieving it with purely volunteer and part-time 
labour are basically zero).



Cheers,

Nick.


-- 

Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] GnuPG signatures on PyPI: why so few?

[Steve Dower]

FWIW, I dropped a portable version into the windows-installer externals that 
are pulled down by the release scripts (from svn.p.o). It does require me to 
import my key on new machines, but since I don't use it for anything but 
re-signing the releases it's worth it to avoid all the intrusions.

So it's definitely possible, just a matter of finding and including the right 
dependencies to copy around.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Paul Moore" 
Sent: ‎3/‎12/‎2017 7:36
To: "Ben Finney" 
Cc: "Distutils" 
Subject: Re: [Distutils] GnuPG signatures on PyPI: why so few?

On 12 March 2017 at 12:13, Ben Finney  wrote:
>
>> As a Windows user, I've "played" with it in the past, and found it
>> frustratingly difficult.
>
> I hope many people here will find the guide published by the FSF, Email
> Self-Defense , a useful walk
> through how to set it up properly.

That's about email, though, and as such irrelevant here. I have no
interest in setting up GPG for my email. Part of what I meant by
"intrusive" was "installs plugins for things like email and file
encryption that I don't want".

Part of my issue here is that people promoting signing tend to think
of it as a way of life, rather than as an annoying little extra step
that is needed for one specific activity (publishing to PyPI in the
context of this thread). There's essentially nothing written from the
POV of "you have no interest in signing, and are only doing it because
someone's insisting that you do - so here's how to do the least
possible to make them shut up". You may not agree with that attitude,
but it is very common in my experience, and documents that start by
trying to change the reader's opinion get discarded *remarkably* fast.

But this is way off-topic, so I'll refrain from saying anything more.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] distlib and wheel metadata

[Steve Dower]

On 23Feb2017 0914, Donald Stufft wrote:



On Feb 23, 2017, at 11:04 AM, Nick Coghlan > wrote:

Redistributors may *ask* a publisher to reclassify their project as a
library or a devtool (and hence also avoid pinning their dependencies
in order to make integration easier), but publishers will always have
the option of saying "No, we want to you to treat it as an
application, and we won't help your end users if we know you're
overriding our pinned dependencies and the issue can't be reproduced
outside your custom configuration".



This whole discussion feels like trying to overcomplicate something
that’s already not a simple to solve a problem that I don’t think is
really that widespread. My estimation is that 99% of people who are
currently using ``==`` will just immediately switch over to using
whatever flag we provide that allows them to still do that. Adding a “do
the thing I asked for” detritus to the project seems like a bad idea.

It’s not really any different than if a project say, only released
Wheels. While we want to encourage projects to release sdists (and to
not ping versions) trying to enforce that isn’t worth the cost. Like
most packaging issues, I think that it’s best solved by opening up
issues on the offending project’s issue tracker.


+1. This has been my feeling the entire time I spent catching up on the 
thread just now.


As soon as "user education" becomes a requirement, we may as well do the 
simplest and least restrictive metadata possible and use the education 
to help people understand the impact of their decisions.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Indexing modules in Python distributions

[Steve Dower]

I'm interested, and potentially in a position to provide funded infrastructure 
for this (though perhaps not as soon as you'd like, since things can move 
slowly at my end).

My personal preference would be to download a full list. This is slow moving 
data that will gzip nicely, and my uses (in IDE) will require many tentative 
queries. I can also see value in a single-query API, but keep it simple - the 
value here is in the data, not the lookup.

As far as updates go, most packaging systems should have some sort of release 
notification or update feed, so the work is likely going to be in hooking up to 
those and turning it into a scan task.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Thomas Kluyver" 
Sent: ‎2/‎7/‎2017 3:30
To: "distutils-sig@python.org" 
Subject: [Distutils] Indexing modules in Python distributions

For a variety of reasons, I would like to build an index of what
modules/packages are contained in which distributions ('packages') on
PyPI. For instance:

- Identifying requirements by static analysis of code: 'import zmq' ->
requires pyzmq
- Finding corresponding packages from different packaging systems: pyzmq
on PyPI corresponds to pyzmq in conda, and python[3]-zmq in Debian
repositories. This is an oversimplification, but importable module names
provide a common basis to compare packages. I'd like a tool that could
pick between different ways of installing a given module.

People often assume that the import name is the same as the name on
PyPI. This is true in the vast majority of cases, but there's no
requirement that they are the same, and there are cases where they're
not - pyzmq is one example.

The metadata field 'Provides' is, according to PEP 314, intended for
this purpose, but the standard packaging tools don't make it easy to
use, and consequently very few packages specify it.

I have started putting together a tool to index wheels. It reads a .whl
file, finds modules inside it, and tries to identify namespace packages.
It's still quite rough, but it worked with the wheels I tried.
https://github.com/takluyver/wheeldex

Is this something that other people are interested in?

One thing I'm trying to work out at the moment is how the data would be
accessed: as a web service that tools can query online, or more like
Linux packaging, where tools download and cache a list to do lookups
locally. Or both? There's also, of course, the question of how the index
would be built and updated.

Thanks,
Thomas
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] RFC: PEP 541 - Package Index Name Retention

[Steve Dower]

On 13Jan2017 1050, Lukasz Langa wrote:

Thanks for review, Steve!



On Jan 13, 2017, at 10:35 AM, Steve Dower <steve.do...@python.org> wrote:

An *abandoned* project can be transferred to a new owner for purposes
of reusing the name when ALL of the following are met:
...


The list here is nearly identical to the previous section


The "skin in the game" behavior is different.


Fair enough. Perhaps we should avoid using the idiom though (as 
suggested earlier) to avoid any potential loss in translation.



I would actually like to be able to name-squat for a period between a project 
being started and being released (particularly in my own context, I often need 
to keep a project private until it has been internally tested/reviewed/scanned 
and the lawyers have signed off, at which point it may require a new review if 
the name has to change).

Presumably for a reachable uploader who can give an explanation, this won't result in the 
immediate loss of the name. But suggesting a time limit may help reduce support requests 
("project is name squatting for at least 6 months" feels okay to me, but not 
wedded to it).


I don't want to suggest arbitrary limits on acceptable name squatting because 
this can be abused. As long as you squat and nobody calls you out on it before 
your first functional release, that's okay. If you squat on a great name and 
somebody comes along with an existing notable project wanting that name, the 
case it rather clear though.


So perhaps name-squatting belongs in the "this project is abandoned and 
I want the name" section rather than the "this project is invalid and 
I'm flagging it via support channels" section? (Or maybe I misunderstood 
the intent of the separate sections, which I'm sure is also useful 
feedback :) )



(As a semi-related aside, I'm currently squatting on the 'microsoft' and 
'windows' packages for trademark protection reasons. They may never get any 
functionality, but that's better than someone else having the name. This sort 
of squatting doesn't necessarily need to be explicitly called out in policy, 
but maybe it's worth a mention?)


I wanted to avoid touching on trademark issues because IANAL.


Very good point. Since nobody directly involved in this policy is a 
lawyer, it might be worth clearly stating what the index maintainers are 
responsible for in the case of a potential legal dispute with an 
unreachable package owner, or one who is deliberately/maliciously making 
themselves unreachable.


Or maybe it's a rare enough case that it doesn't matter? We certainly 
resolved our last issue easily enough, though it did require the index 
maintainers to put us in direct contact with the package owner. Maybe 
stating that "the index maintainers are not responsible for evaluating 
the legal status of intellectual property, and may only establish direct 
contact between the complainant and a reachable package owner with 
mutual consent" is the way to go? (And get VanL to sign off on the 
wording, just in case there's some oddity here I'm not aware of.)


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Announcement: TLSv1.2 will become mandatory in thefuture

[Steve Dower]

"I don’t think it’s a particularly big deal to tie the tls module to the Python 
lifecycle though"

I'd hope that the API of this module is stable enough and the native part of 
the implementation is OS-specific enough that we may not even need to update it 
that often. (I'm advocating very strongly for just using the OS APIs to 
implement it, and those don't change often enough for us to need to worry.)

The Linux builds can link to OpenSSL, but there shouldn't be anything requiring 
OpenSSL for this module, so the update timeframe is totally different. But I've 
now joined security-sig, which is where the discussion seems to be, so I'll 
stop designing things here :)

Top-posted from my Windows Phone

-Original Message-
From: "Donald Stufft" 
Sent: ‎1/‎11/‎2017 19:48
To: "Nick Coghlan" 
Cc: "DistUtils mailing list" 
Subject: Re: [Distutils] Announcement: TLSv1.2 will become mandatory in 
thefuture



On Jan 11, 2017, at 10:40 PM, Nick Coghlan  wrote:


On 12 January 2017 at 13:00, Donald Stufft  wrote:

This doesn’t work well because it’s not something that pip is going to be
able to upgrade on Windows, because the .so will be locked when pip imports
it on Windows and we won’t be able to uninstall it to do an upgrade. We had
to disable the automatic use of pyOpenSSL for this reason too. The only C
stuff that pip can reliably use is the standard library.


Ugh, I'd completely forgotten about that limitation of Windows filesystems.

And the main alternatives I can think of involve copying files around
as pip starts up, which would be unacceptably slow for a command line
app :(




I don’t think it’s a particularly big deal to tie the tls module to the Python 
lifecycle though, we’ve got a precident for PEPs that backport important 
security critical stuff and most things are presumably going to be things that 
we don’t really even need a backport or a PEP for (I’m thinking things like 
ciphers and such). Particularly if this new thing is documented up front 
clearly what things you can depend on for compatibility (api and such) and what 
things can change in minor releases (keeping up with the security joneses 
stuff).


I think the big thing that really killed the ssl module for so long in Python 
was the 2.x vs 3.x split with 2.7 living for a _very_ long time, and then no 
culture of back porting security important changes to it.



—
Donald Stufft___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Steve Dower]

"Alternatively, I've recently started using Visual Studio Code as my editor for 
work ..."

FWIW, the long-term Python story in VSCode is currently (largely) one of my 
responsibilities, and that bootstrapping flow is exactly one of the pieces I 
really want to put in. Unfortunately, nobody has let me have any engineers to 
implement it yet :( (mostly for big-company-politics reasons, rather than 
hiring trouble)

Top-posted from my Windows Phone

-Original Message-
From: "Nick Coghlan" 
Sent: ‎12/‎16/‎2016 5:08
To: "Glyph Lefkowitz" 
Cc: "Barry Warsaw" ; "DistUtils mailing list" 

Subject: Re: [Distutils] Maintaining a curated set of Python packages

On 16 December 2016 at 20:57, Glyph Lefkowitz  wrote:



Anyhow, Xcode is far from perfect - many of the places it touches the UNIX 
pipeline are extremely sharp edges you can easily impale yourself on (and don't 
get me started about codesigning) - but it nevertheless points at a different 
potential direction.  For example; why expose the concept of a "virtual 
environment" directly at all?  "New Project" could just create a 
requirements.txt and a setup.py for you, alongside a git repo and a virtualenv 
for that project.  Or, the UI could be geared towards setting up a tox.ini 
rather than a virtualenv, and run everything through tox so it's in an isolated 
environment with defined requirements.  This is a best practice anyway so why 
not make it easier to start early?


This might all be way too much work, but I think it's important to remember 
it's possible.


Yeah, I think we agree more than we disagree here. The main thing is that one 
of the key ways newcomer-friendly environments make themselves more 
approachable is to *constrain choice*.


XCode usability benefits from being Apple-centric. Ditto for Visual Studio and 
MS.


Linux and Python, by contrast, were both born out of a DIY culture where folks 
being free to choose their own tools was initially perceived solely as a highly 
desirable feature, rather than as a potential barrier to entry for newcomers.


That means there's an argument to be made that something like YHat's Rodeo [1] 
might be a better starting point for data analytics in Python than jumping 
straight to Jupyter Notebook, and it's also why the Mu editor [2] exists as a 
dedicated tool for folks learning Python by way of the micro:bit project.

[1] http://rodeo.yhat.com/docs/

[2] http://codewith.mu/

However, the reason I brought up the Curse and Firefox GUI examples was to 
emphasise the problems they hide from the default rich client experience:


- their default focus is on managing one environment per device



In the analogous Python tool, one could replace "per device" with "per project" 
- and perhaps have a "default project" so something useful could happen even 
before you've decided what you're doing...


But we've immediately bumped the complexity level up in doing so, and it's a 
level of complexity that many people initially spending all of their 
development time on a single project may not need. 


I thought this thread was already interminable, I look forward to reading the 
never-ending rest of it now that you've raised the grim spectre of the PyPI 
user-ratings feature from the dead :).


All the arguments against integrating user ratings into a service that's 
focused on lowering barriers to publication still hold, so I'm really just 
noting that that decision to create a friendlier publishing environment *does* 
introduce some additional constraints elsewhere in the distribution pipeline.

User-curated package sets strikes me as the _lowest_ priority feature out of 
all of those, if we are ordering by priority to deliver a good user experience. 
 I know "steam curators" have been brought up before - but we're talking about 
adding curators (one of my least favorite features of Steam, for what it's 
worth) before we've added "install game" ;-).


In many educational contexts, adding "install game" without support for 
institutional curators of some kind is a complete non-starter (even if those 
curators are a collaborative community like a Linux distribution, there's still 
more accountability than software publishing sites like PyPI tend to provide).



I initially wanted to disagree when I read this, but I'm not actually sure what 
educational contexts you're talking about, and why "accountability" is 
important?


Schools, mainly. Lots of administrators are still scared of the internet, so 
one of the attractions of things like Raspberry Pi is that the software updates 
come from Debian rather than directly from the software publishers.


Sometimes you can get away with "What the bureaucracy doesn't know won't hurt 
it", but it's more convenient when teachers don't have to do that.

 
"beginner" is a direction, and not a fixed position; many people more 
"beginner" than the current 

Re: [Distutils] Maintaining a curated set of Python packages

[Steve Dower]

The "curated package sets" on PyPI idea sounds a bit like Steam's curator 
lists, which I like to think of as Twitter for game reviews. You can follow a 
curator to see their comments on particular games, and the most popular 
curators have their comments appear on the actual listings too.

Might be interesting to see how something like that worked for PyPI, though the 
initial investment is pretty high. (It doesn't solve the coherent bundle 
problem either, just the discovery of good libraries problem.)

Top-posted from my Windows Phone

-Original Message-
From: "Donald Stufft" 
Sent: ‎12/‎15/‎2016 4:21
To: "Freddy Rietdijk" 
Cc: "DistUtils mailing list" ; "Barry Warsaw" 

Subject: Re: [Distutils] Maintaining a curated set of Python packages



On Dec 15, 2016, at 7:13 AM, Freddy Rietdijk  wrote:


> Putting the conclusion first, I do see value in better publicising

> "Recommended libraries" based on some automated criteria like:


Yes, we should recommend third-party libraries in a trusted place like the 
documentation of CPython. The amount of packages that are available can be 
overwhelming. Yet, defining a set of packages that are recommended, and perhaps 
working together, is still far from defining an exact set of packages that are 
known to work together, something which I proposed here.




We could theoretically bake this into PyPI itself, though I’m not sure if that 
makes sense.


We could also probably bake something like “curated package sets” into PyPI 
where individual users (or groups of users) can create their own view of PyPI 
for people to use, while still relying on all of the infrastructure of PyPI. 
Although I’m not sure that makes any sense either.


—
Donald Stufft___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] 3.6.0b2 on windows

[Steve Dower]

On 12Oct2016 0418, Robin Becker wrote:

I have a question regarding an error message I see when building an
extension for windows using the 3.6.0b2 release



C:\ux\XB33\py36_x86\lib\site-packages\wheel\pep425tags.py:77:

RuntimeWarning: Config variable 'Py_DEBUG' is unset, Pytho

n ABI tag may be incorrect
  warn=(impl == 'cp')):
C:\ux\XB33\py36_x86\lib\site-packages\wheel\pep425tags.py:81:

RuntimeWarning: Config variable 'WITH_PYMALLOC' is unset,

Python ABI tag may be incorrect
  warn=(impl == 'cp')):


is this something I need to worry about?


IIRC, these warnings have been suppressed in a newer version of pip 
(possibly not released yet?). They are harmless on Windows.


We do need to update the bundled version of pip in 3.6, preferably 
before b3, to pick up the fixes to distlib and the fix for these warnings.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Module Installation Issues

[Steve Dower]

On 13Sep2016 1559, Matthew Brett wrote:

Perhaps a better idea would be to add some smarts to the REPL (but not to 
Python itself) that would detect something like:


pip install


And print a better error message that gives a better indication about what’s 
gone wrong besides a SyntaxError?


I was thinking the same thing.  If it could also catch accidental


python3 -m pip install


commands, that would be even better.


I feel like these would deserve text adventure style errors:

>>> pip install spam
I don't see a pip here

>>> python
You're already running a Python!

>>> pip uninstall eggs
You ask nicely, but pip refuses to uninstall from inside this shell.

Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Module Installation Issues

[Steve Dower]

On 13Sep2016 1555, Donald Stufft wrote:



On Sep 13, 2016, at 6:41 PM, Steve Dower <steve.do...@python.org> wrote:

I think it's one of these things where we should suck it up and let the 90% 
case work fine, then display a big fat warning if anything weird may have 
happened and let users sort it out themselves.



I am unsure. One of the really egregious and hard to debug weirdness is going 
to be something like:



import foo.bar  # foo, and foo.bar are in sys.modules
pip.install(“thing”)  # This implicitly upgrades foo
import foo.widget  # the old foo is in sys.modules, but the new foo.widget.




Except my version of this goes more like:

>>> import foo.bar  # foo, and foo.bar are in sys.modules
>>> pip.install("thing")  # This implicitly upgrades foo
WARNING: Packages were updated. You need to restart Python now.
>>> import foo.widget  # because I happily ignore messages starting 
with WARNING


Doesn't matter to me whether it's "safe" to keep going or not - if any 
files are overwritten at all then show the warning. (This is exactly the 
argument I was expecting when I said "those who want it to be perfect in 
every scenario" ;) )


Cheers,
Steve

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Module Installation Issues

[Steve Dower]

On 13Sep2016 1500, Paul Moore wrote:

On 13 September 2016 at 21:12, Thomas Kluyver  wrote:

One thing I'd quite like to see Python grow is a standard function to
install packages from inside Python.


That's not too hard in principle - pip.main(['install', package]) is
basically all you'd need (modulo various design details, and wrapping
it in a nice user friendly function). The bigger problems are:

1. pip is an independent module (albeit shipped with Python) and it's
not clear to what extent Python would want a builtin function that
depends on a non-stdlib module. But that's not a technical issue.
2. Python's module system isn't really designed around installing new
modules while a process is running. There's caching involved, so the
effects can be unpredictable. It's difficult to know whether the
benefits (avoiding confused users who tried to install from a Python
prompt) would outweigh the costs (confused users having installed a
module but not being able to see it because the module system didn't
notice things had changed).

I'm not honestly sure how big the "installing while a process is
running" issue would be - I did a few simple experiments and couldn't
immediately trigger weirdness, but I believe it can happen. And things
get significantly worse if we allow upgrades from the Python prompt
rather than just installs (e.g., if you have already imported
something from the old version and then upgrade).

Overall, it'd probably be a nice thing to have, but it's nowhere near
as simple as it seems at first glance.
Paul


I think it's one of these things where we should suck it up and let the 
90% case work fine, then display a big fat warning if anything weird may 
have happened and let users sort it out themselves.


A simple builtin that tries to run "sys.executable -m pip {args}" in a 
subprocess and displays "success", "success but you should probably 
restart Python now", "failed because you don't have pip", "failed and 
have all the output" is going to make lots of people happy, even if it 
upsets those who want it to be perfect in every scenario.


Cheers,
Steve

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Module Installation Issues

[Steve Dower]

On 13Sep2016 1312, Thomas Kluyver wrote:

On Tue, Sep 13, 2016, at 08:39 PM, Steve Dower wrote:

It would help if you could post the full error output (sanitizing
paths if needed). But you may just need to upgrade pip (python -m
install -U pip).


I think Ryan may have typed that command at a Python prompt rather than
a system command prompt. Unfortunately the distinction often isn't clear
in examples, because the experienced developers writing the instructions
are used to guessing which commands are Python and which are system
commands.

One thing I'd quite like to see Python grow is a standard function to
install packages from inside Python. In R, the install.packages()
function is the default way to get third party packages, and I think
staying in one interactive prompt does make things easier for new users
to understand.


Ah, I suspect you're right.

I have considered adding a "Python x.y Shell" start menu item that would 
configure PATH properly for commands like "python" and "pip". 
Instinctively, it seems this would be more useful than a direct shortcut 
to the executable, but at the same time I don't want to start competing 
with all the other app-specific shells out there.


Cheers,
Steve


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Module Installation Issues

[Steve Dower]

It would help if you could post the full error output (sanitizing paths if 
needed). But you may just need to upgrade pip (python -m install -U pip).

Knowing exactly where the syntax error is coming from will help us figure out 
which package has the problem. There are at least three involved here.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Mills, Ryan" 
Sent: ‎9/‎13/‎2016 9:56
To: "distutils-sig@python.org" 
Subject: [Distutils] Module Installation Issues

I just recently downloaded Python 3.5 and cannot seem to install any packages 
like Numpy, etc.  I have tried all the instructions on the website and keep 
getting errors:
 
For example, when I type “python –m pip install Numpy” it returns  a Syntax 
Error. I am completely new to Python so I must be missing something here – I 
haven’t altered any files since installing it the other day. Do I use the 
Python IDLE Shell? Are there other packages I need to install first?  Any help 
would be greatly appreciated.  
 
-Ryan
 
Ryan Mills
Quantitative Risk Analyst, Banking Officer

Capital Analytics & Stress Testing
2000 McKinney Avenue, Suite 700
Dallas, TX 75201
214.932.6653 direct
20.6653 internal
ryan.mi...@texascapitalbank.com



 
If you are not the addressee and have received this email in error, please 
notify me immediately. This email is confidential and may contain privileged or 
proprietary information that is unlawful for you to read, copy, distribute, 
disclose or otherwise use in any way.

 
 ___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What is the official position on distutils?

[Steve Dower]

"add it to setuptools first, and then add things to distutils where we feel 
they're sufficiently stable to not need the benefit of the faster setuptools 
update cycle"

I nominate this as the official policy on API changes for distutils, with 
regular maintenance mode policies applying to anything else.

Top-posted from my Windows Phone

-Original Message-
From: "Nick Coghlan" 
Sent: ‎9/‎4/‎2016 2:19
To: "Sylvain Corlay" 
Cc: "distutils sig" 
Subject: Re: [Distutils] What is the official position on distutils?

On 4 September 2016 at 06:44, Sylvain Corlay  wrote:
> Hi Brett,
>
> On Sat, Sep 3, 2016 at 8:05 PM, Brett Cannon  wrote:
>>
>>
>> If Jason is up for the responsibility that seems like a reasonable
>> approach to take. It also helps test out features in setuptools first before
>> upstreaming it.
>>
>
> How do you see `has_flag` get into setuptools? By monkey-patching distutils'
> ccompiler to have it aside of `has_function` when setuptools is imported?
>
> I find really weird the idea of adding this in a convoluted fashion instead
> of allowing incremental improvement of distutils.

The change to distutils would still be a plain patch to distutils, it
would just be accepted at the API design level in setuptools first.

The problem you're running into right now isn't a technical one - it's
that there isn't anyone that currently feels like they have sufficient
design authority over the distutils API to accept your proposal, hence
Brett starting this thread to address that underlying recurring
question, rather than the specifics of your change.

Jason *definitely* has that design authority over setuptools though,
and will be tasked with making any API additions available on older
versions of Python via setuptools regardless of what policy we adopt
for distutils maintenance, so if he's amenable to the idea, it makes
sense to me to invert the order we ask those questions: add it to
setuptools first, and then add things to distutils where we feel
they're sufficiently stable to not need the benefit of the faster
setuptools update cycle.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Proposed new Distutils API for compiler flagdetection (Issue26689)

[Steve Dower]

Some of the core development team will be sprinting full time for the week 
leading up to beta 1, so expect a lot of things to get added then.

My main concern with this is compatibility rather than the interface, but as a 
new feature I think it's best to be only in setuptools. Distutils is minimal 
support for building and as long as it works I don't see any reason to change 
it given the ease of getting alternatives.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Sylvain Corlay" 
Sent: ‎8/‎28/‎2016 3:35
To: "Ralf Gommers" 
Cc: "DistUtils mailing list" 
Subject: Re: [Distutils] Proposed new Distutils API for compiler flagdetection 
(Issue26689)

Hi All,


The beta deadline for new features is approaching dangerously.


I agree with Thomas that being able to require Python 3.6 for a project does 
not appear so distant for me (as soon as it is a Python 3 project only).


Any chance to get this through? Checking support for language features will be 
more and more required since new version of the C++ standard are becoming more 
frequent. I understand that it is not an issue for a project like numpy, but 
this is a check I do in every single one of my extension projects.

Thanks,


Sylvain



On Thu, Aug 25, 2016 at 6:41 AM, Ralf Gommers  wrote:



On Mon, Aug 22, 2016 at 6:50 PM, Thomas Kluyver  wrote:

There's obviously some cost in code duplication; I haven't looked at the code 
in question, so I don't know how bad this is.



This patch is pretty short and understandable, so not bad.




I've run into this argument before when trying to change things in 
non-packaging-related parts of the stdlib, and I agree with Sylvain that it's 
fundamentally problematic. If we're trying to improve the stdlib, we're 
obviously taking a long view, but that's how we ensure the stdlib is still 
useful in a few years time. This goes for packaging tools as much as anything 
else.



This I don't agree with - packaging is fundamentally different for the reasons 
Donald gave.


Ralf




I already have projects where I'm happy to require Python >=3.4, so being able 
to depend on Python 3.6 is not such a distant prospect.



Thomas___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Steve Dower]

On 23Aug2016 0937, Brett Cannon wrote:

I should also mention I have never come across anyone at Microsoft use
the bdist_msi or bdist_winst installers (I've added Steve to this email
in case my experience is wrong).


In large part this is because I've gotten to them first :)

Personally I don't like bdist_msi or bdist_winst as neither of them 
correctly sets dependencies on the underlying Python install, so 
removing Python does not remove the package. But I can see why people 
may prefer to use them.


I see no harm in not installing them via pip though. They should be 
downloadable "associated files" at minimum, and I don't have any opinion 
about making them available via index data (except that installers 
should be free to ignore them).


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions onPyPI?

[Steve Dower]

We could also extend the py.exe launcher to resolve a matching Python install 
for a wheel and run pip to install it. Then double-clicking a wheel file would 
do something useful.

Having a standard UI would be better, though non-trivial.

Top-posted from my Windows Phone

-Original Message-
From: "Paul Moore" 
Sent: ‎8/‎23/‎2016 7:51
To: "Antoine Pitrou" 
Cc: "Distutils" 
Subject: Re: [Distutils] Deprecating little used file types/extensions onPyPI?

On 23 August 2016 at 15:12, Antoine Pitrou  wrote:
> On Tue, 23 Aug 2016 10:36:35 +0100
> Paul Moore  wrote:
>>
>> I don't see any sign of *anyone* working on a curated distribution for
>> Windows along the lines of Linux distros or Homebrew. (Unless you
>> count cross-platform stacks like conda, which IMO are a different
>> scenario than "system" Python installs).
>
> Under Windows, there isn't much moral difference between a conda install
> (really a Miniconda or Anaconda install) and a python.org Python
> install.  You can even install Anaconda or Miniconda system-wide.
>
> (Miniconda is a minimal install of Python + conda, while Anaconda is a
> pretty large selection of packages in addition - though not the entire
> official repo contents, and not counting community packages)

Agreed - sorry, I was responding more to Nick's implication around
"system" management of Python, conda already has a well-established
management process, but AIUI, it's largely independent of PyPI/pip
(although it interoperates with those). So in the context of Windows
package managers, conda is no more relevant there than it is in
relation to (say) rpm or deb.

Realistically, though, I'd expect people wanting a Python stack on
Windows to gravitate more and more towards distributions like Anaconda
(maybe less so outside its core area of scientific use) with the
remaining users sticking to the python.org builds and pip.

However, I'm not that keen on continuing support for bdist_wininst and
bdist_msi. I'd rather put the effort into making pip and wheels a
compelling solution for such users. IMO, it already is for command
line use (the more so as more hard to build packages start providing
wheels). The place where pip falls down is for people who want to
stick to a GUI. For those people, I believe Idle is likely to be
getting an interface to pip, and I'd like to see other tools like PTVS
and PyCharm get similar capabilities (if they don't have them
already). That may mean exposing an interface to pip's install
mechanisms that's better than "use subprocess to call pip", but we'd
need some concrete use cases to work out the best form of such
interfaces.

On the matter of Chocolatey, it should be pretty trivial to create
recipes for installing Python packages via pip. So I'd see pip+wheel
as remaining the standard interface pretty much indefinitely.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions onPyPI?

[Steve Dower]

Do you mean like zipapp and *.pyz files?

Top-posted from my Windows Phone

-Original Message-
From: "Nick Coghlan" 
Sent: ‎8/‎16/‎2016 22:42
To: "Leonardo Rochael Almeida" 
Cc: "distutils sig" 
Subject: Re: [Distutils] Deprecating little used file types/extensions onPyPI?

On 17 August 2016 at 15:21, Nick Coghlan  wrote:
> While rebranding eggs (or a close derivative) as "pypath entries"
> could help make them more self-explanatory (especially when it comes
> to explaining how the zipped form differs from wheel files), doing so
> wouldn't actually give us any more architectural flexibility in
> practice than the last option.

That said, if folks *did* want to go down this path, then an important
use case to consider is "single archive applications" suitable for use
in services like AWS Lambda and Azure Cloud Functions, where you get a
language runtime to play with, but need to provide everything you want
to run in that environment as a single pre-bundled archive. Similar
problems also show up when building rich client applications for
Windows, Mac OS X, Android and iOS.

This means you want tooling that targets the direct dependency
bundling model, rather than the runtime environment configuration
model favoured by pip and virtualenv.

So this particular challenge *shouldn't* be framed as just a backwards
compatibility and migration concern for buildout et al - the "all
inclusive application bundle" approach is still an important use case,
even though it isn't the lowest common denominator cross-platform
dependency specification, retrieval and installation problem handled
by the default toolchain.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] msvc9compiler.py - vcvarsall.bat is living onefolder higher - VS90COMNTOOLS

[Steve Dower]

There may be a bug in setuptools. We've already seen one reported and fixed in 
the last week, so possibly this is related.

Issues in setuptools go to their github page (don't have the link handy on my 
phone). Nothing has changed in distutils in over a year here and it's been 
working just fine, so I'd look at what they're doing in setuptools.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "posti...@andreaskrueger.de" 
Sent: ‎8/‎10/‎2016 6:34
To: "distutils-sig@python.org" 
Subject: Re: [Distutils] msvc9compiler.py - vcvarsall.bat is living onefolder 
higher - VS90COMNTOOLS



Hi Steve,
Thanks for your superfast answer.


> There's actually also a requirement for a relatively recent version (>=6.0) 
> of setuptools
>  in order to build correctly using the Visual C++ Compiler for Python. 
This is the installed version

 setuptools: 25.1.6-py27_0
and 25>6 ?




This will allow the compiler to be found without using the environment 
variable. 


Then perhaps consider to completely remove/deprecate the route via environment 
variable? 
Because there are tons of workarounds & manuals out there which actually point 
towards using the environment variable.

Or fix it.




I suggest ensuring that setuptools is up to date

So the setuptools included in the newest anaconda is not new enough?




, and either import it in the setup.py (you don't have to do anything except 
import it) or use pip to install your packages. 

Not sure I understand. 
I have been installing mc-stan.org - via 'conda install -c patricksnape 
pystan'. It worked only after (solving other problems, and) patching your 
distutils. ___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] msvc9compiler.py - vcvarsall.bat is living one folder higher - VS90COMNTOOLS

[Steve Dower]

On 09Aug2016 1519, Dr. Andreas Krueger wrote:

I could not find a github repo, otherwise I would have filed this as an
"issue" there.


The subfolder "VC" is wrong in msvc9compiler.py

- at least on my system (win7-64bit, py2.7.12, anaconda 2.4.1, conda 4.1.11)


So in your "msvc9compiler.py"

instead of
productdir = os.path.join(toolsdir, os.pardir, os.pardir, "VC")

it needs to be
productdir = os.path.join(toolsdir, os.pardir, os.pardir)


Why do I say that?
When googling the error message:  "DistutilsPlatformError: Unable to
find vcvarsall.bat", it is recommended "everywhere" on the web, to
download "VCForPython27.msi", Microsoft Visual C++ Compiler for Python
2.7  http://aka.ms/vcpython27


BUT then

vcvarsall.bat

ends up being in

"C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++
for Python\9.0\vcvarsall.bat"

and NOT in

"C:\Users\Andreas\AppData\Local\Programs\Common\Microsoft\Visual C++
for Python\9.0\VC\vcvarsall.bat"

where YOUR code is expecting it.


Perhaps you actually want to check BOTH places, to keep compatibility
with some older versions / different setups?
Because I suppose, your code had worked, at some stage in the past.


Thanks a lot!
   Andreas



There's actually also a requirement for a relatively recent version 
(>=6.0) of setuptools in order to build correctly using the Visual C++ 
Compiler for Python. This will allow the compiler to be found without 
using the environment variable.


I suggest ensuring that setuptools is up to date, and either import it 
in the setup.py (you don't have to do anything except import it) or use 
pip to install your packages.


Cheers,
Steve


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setuptools 25.1.6 broke numpy.distutils on Windows

[Steve Dower]

On 09Aug2016 0213, Antoine Pitrou wrote:

Just a heads-up that latest setuptools is incompatible with
numpy.distutils on Windows:
https://github.com/pypa/setuptools/issues/728


For those who don't want to dig into the issues and pull requests, it 
seems that multiple libraries were patching different parts of distutils 
and adding extra quotes to the command line. This was apparently only 
occurring with Python 3.5.


The main reason I want to add to this post is that *Python 3.5 is still 
taking bugfixes*. If there is a problem in distutils, please report it 
to bugs.python.org first so we can fix it, and then look into 
monkeypatching the older versions.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] cffi & Py_LIMITED_API

[Steve Dower]

On 01Aug2016 0702, Nick Coghlan wrote:

On 1 August 2016 at 23:36, Daniel Holth  wrote:

build_ext command determines
the DLL extension. It could be patched or modified to read an "I'm ABI3"
flag on the Extension() object.

We could pass an ABI3 flag to bdist_wheel in the same way we ask for
universal 'py2.py3-none-any'. To be set if the wheel contained only ABI3
extensions, and ignored on py2.


The general idea sounds good to me, but as a slight bikeshed on the
flag name, perhaps "cpabi3"?

That's a mash-up of the 'cp' interpreter code for CPython, with the
'abi3' stable ABI tag.

Longer term, we may want to allow people to version that (as new APIs
may sometimes be added to the stable ABI, which you gain access to at
the C level by setting Py_LIMITED_API to the corresponding CPython hex
version rather than just defining it [1]), but as a starting point
enabling access to the initial 3.2 stable ABI used by cffi should be
sufficient.


The DLL tag on Windows will have to just be ".pyd" if you want to 
support back prior to 3.5. In 3.5 you can use ".cp35-win32.pyd" or 
".cp35-win_amd64.pyd" and the importer will prefer that DLL over a plain 
".pyd", but my proposal to also support ".cp3-${PLAT}.pyd" here didn't 
make it.


The wheel tag is more important than the DLL tag. (Possibly you're not 
even suggesting changing the DLL name at all and just passing the flag 
through for the build option? Hard to tell from the discussion.)


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Contributing money to package authors/maintainers via PyPI

[Steve Dower]

On 23Jul2016 1211, Donald Stufft wrote:



On Jul 23, 2016, at 2:40 PM, Nicholas Chammas
> wrote:

I know a more concrete proposal would have to address a lot of details
(e.g. like how to split contributions across multiple maintainers),
and perhaps there is no way to find the resources to build or maintain
such a thing in the first place. But just for now I’d like to separate
essence of idea from the practical concerns of implementing it.



I’m mulling over the idea in my head, but one other thing we’d need to
figure out is the *legality* of doing this and if it’s something the PSF
is willing to do at all.


Maybe it's as simple as a user profile "Donations URL" field that is 
also listed on any packages owned by that user? Or possibly a 
per-package URL? (The latter may work out better for people who manage 
both personal and work packages, which is currently the system we're 
using at Microsoft until we get a centralised publishing system set up.)


Cheers,
Steve

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Steve Dower]

I forget the exact names but there's a range of SQL Server packages that also 
fit in here. Perhaps I get to hear more complaints about those because of where 
I work :)

But you're right, it may be a small enough problem to handle it that way.

Top-posted from my Windows Phone

-Original Message-
From: "Donald Stufft" <don...@stufft.io>
Sent: ‎7/‎14/‎2016 17:25
To: "Steve Dower" <steve.do...@python.org>
Cc: "Daniel D. Beck" <dan...@ddbeck.com>; "distutils-sig" 
<distutils-sig@python.org>
Subject: Re: [Distutils] Outdated packages on pypi



On Jul 14, 2016, at 8:19 PM, Steve Dower <steve.do...@python.org> wrote:


I'm still keen to find a way to redirect people to useful forks or alternative 
packages that doesn't require thousands of mentions at conferences for all time 
ala PIL.




I’m not opposed to this but we’ll want to make sure we’re careful about how we 
do it. PIL is an easy example where the maintainer is gone and there is a 
community fork of it. But I struggle to come up with very many more examples of 
this where there is something that is:


- Popular enough that enough people are tripping over it to make it worth it.
- There is a clear successor (or successors).


Off the top of my head I can only really think of PIL, and *maybe* suds. Unless 
there’s a lot of these maybe all we really need is a policy for when 
administrators can/will edit the page to direct people towards a different 
project or a way to add an admin message directing people to another project.

—
Donald Stufft___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Steve Dower]

Sure, if it's been tried before and people couldn't control themselves back 
then (before my time too, and the internet wasn't as blatantly toxic six+ years 
ago as it is now) then that's reason enough not to try again.

I'm still keen to find a way to redirect people to useful forks or alternative 
packages that doesn't require thousands of mentions at conferences for all time 
ala PIL.

Top-posted from my Windows Phone

-Original Message-
From: "Donald Stufft" <don...@stufft.io>
Sent: ‎7/‎14/‎2016 17:06
To: "Steve Dower" <steve.do...@python.org>
Cc: "Daniel D. Beck" <dan...@ddbeck.com>; "distutils-sig" 
<distutils-sig@python.org>
Subject: Re: [Distutils] Outdated packages on pypi


> On Jul 14, 2016, at 6:51 PM, Steve Dower <steve.do...@python.org> wrote:
> 
> On 14Jul2016 0619, Daniel D. Beck wrote:
>> Free-form, user-generated content on PyPI would become a pathway for
>> harassment and abuse. Introducing user-generated content on PyPI would
>> necessarily put an emotional burden on package maintainers in addition
>> to the maintenance burden (unless PyPI moderators are going to screen
>> content before maintainers and users see it—given the dearth of
>> resources for PyPI as it is, this strikes me as exceedingly unlikely).
> 
> This is why I listed a set of restrictions to help prevent that:
> 
> * 140 chars (flexible, but short enough to prevent rants)
> * users must be logged in
> * no external links
> * maintainers can delete/dispute comments
> * clear comments on each new release
> * one comment per user per package (implied, but I didn't explicitly call it 
> out in my previous email)
> 
> Do you really think this will be worse than the current state, where abusers 
> *only* have access Twitter, github, reddit and email to harass package 
> maintainers?
> 
> Assuming harassment is not going to be a problem, is there value in letting 
> people add comments directly on the page where users seem to keep ending up?
> 

I don’t believe you can assume that harassment is not going to be a problem.

There’s a fundamental power dynamic here, where publishing your project to PyPI 
is not *entirely* optional. It’s optional in the sense that nobody is going to 
force you to publish your project there, but it’s hard to interact fully with 
the Python ecosystem as a whole for your project if you don’t at least add an 
entry for it there. Given that we have this dynamic, we need to be particularly 
careful how the features we add can be used against people, particularly 
against the most vulnerable people in our community.

We sadly live in a world where our industry is incredibly toxic to, well 
basically everyone but white guys and are actively hostile towards efforts to 
seeing a community become more inclusive. These are people who will regularly 
create multiple twitter accounts in order to spam harassment at people (in 140 
characters) to get around cases where the person has blocked them. These are 
people who will flood comments on GitHub issue trackers for projects they don’t 
even use to bitch about someone changing some pronouns to be more inclusive.

Consider that a rude comment can completely crush someone’s motivation to learn 
Python, or to maintain a package. It can make our community seem all that more 
hostile and I don’t think the vast majority of comments are going to actually 
be very useful. I suspect they will largely be used as yet another support 
venue for random users who are confused (and deleting them doesn’t help those 
users either).

We had a comments and review system years ago (before my time TBH) and the 
backlash against it was so great that it was a major point of contention on 
catalog-sig where Package authors wanted it to be gotten rid of and the 
maintainers at the time pushing back to keep it. We (obviously) eventually got 
rid of it, and I think that is pretty indicative of the idea in general.

Any sort of user created content requires us, the people running PyPI, to 
moderate to some degree. We have to do it now with people who create projects 
with vulgar or offensive names and I don’t believe that we have the man power 
available to us to moderate the comments of a much larger feature that is going 
to incentivize people to make negative comments (and let’s be real, 95% of the 
comments are going to be negative, people rarely reach out to say they’re happy 
but they’re always ready to complain).

This is a lot of words to say that I would be very against this kind of feature 
on PyPI. I am not *entirely* against some sort of automated marker for possibly 
unmaintained packages, but even that I’m sketchy on. Allowing people to poop 
their own content onto project pages for a project they don’t own is just not 
tenable I think.

—
Donald Stufft



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Steve Dower]

On 14Jul2016 0619, Daniel D. Beck wrote:

Free-form, user-generated content on PyPI would become a pathway for
harassment and abuse. Introducing user-generated content on PyPI would
necessarily put an emotional burden on package maintainers in addition
to the maintenance burden (unless PyPI moderators are going to screen
content before maintainers and users see it—given the dearth of
resources for PyPI as it is, this strikes me as exceedingly unlikely).


This is why I listed a set of restrictions to help prevent that:

* 140 chars (flexible, but short enough to prevent rants)
* users must be logged in
* no external links
* maintainers can delete/dispute comments
* clear comments on each new release
* one comment per user per package (implied, but I didn't explicitly 
call it out in my previous email)


Do you really think this will be worse than the current state, where 
abusers *only* have access Twitter, github, reddit and email to harass 
package maintainers?


Assuming harassment is not going to be a problem, is there value in 
letting people add comments directly on the page where users seem to 
keep ending up?


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Steve Dower]

On 13Jul2016 1456, Glyph Lefkowitz wrote:



On Jul 13, 2016, at 1:54 PM, Steve Dower <steve.do...@python.org
<mailto:steve.do...@python.org>> wrote:

Possibly such user-contributed content would be valuable anyway


https://alternativeto.net but for PyPI? :)


Or just more general reviews/warnings/info. "Doesn't work with 
IronPython", "Works fine on 3.5 even though it doesn't say so", etc.


Restrict it to 140 chars, signed in users, only allow linking to other 
PyPI packages, let the maintainer delete comments (or mark them as 
disputed) and I think you'd avoid abuse (or rants/detailed bug 
reports/etc.). Maybe automatically clear all comments on each new 
release as well.


Doesn't have to be complicated and fancy - just enough that users can 
help each other when maintainers disappear.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Steve Dower]

On 13Jul2016 1252, Glyph Lefkowitz wrote:

The primary thing would be to have a banner on the page and a warning
from `pip install´.  Those of us close to the heart of the Python
community already have various ways of reading the tea leaves to know
that things are likely to be unmaintained or bitrotting; the main
purpose of such a feature would be to have an automated way for people
who /don't/ personally know all the prominent package authors and see
them at conferences and meetups all the time to get this information.
 For example: nobody should be using PIL, they should be using pillow.
 Yet there's no way for a new user to figure this out by just looking
at https://pypi.io/project/PIL/ :).

I think that the adjudication process for stealing a name from an
existing owner is something that still bears discussion, but separately.
 Whatever that process is, you'd have to go through it fully after a
package becomes thusly "abandoned", and for the reasons you cite, it
absolutely should not be automated.  Perhaps it shouldn't even be the
way to deal with it - maybe the most you should be able to do in this
case is to expand the "this is unmaintained" warning with a pointer to a
different replacement name.


I like this. Maybe if a maintainer doesn't trigger the switch/publish 
anything for a year, a banner appears on the page with a publicly 
editable (votable?) list of alternative packages - thinking something 
similar to a reviews system with an "I found this review helpful" button.


Possibly such user-contributed content would be valuable anyway, but the 
"probably abandoned" state just moves it to the top of the page instead 
of the bottom.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PyPI index workaround

[Steve Dower]

I'm also interested (for the same support in Visual Studio) though we're 
unaffected by this change.

A batch API to get info for many packages would be great. Currently we scrape 
simple and then post JSON queries for individual packages.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Михаил Голубев" 
Sent: ‎7/‎13/‎2016 13:04
To: "Donald Stufft" 
Cc: "distutils-sig@python.org" 
Subject: Re: [Distutils] PyPI index workaround

I'm sorry, I should have posted my commentary here, not in the separate thread.
 
We have some issues with suggested "/simple" endpoint. Despite the need to 
scrap the web page, old endpoint allowed us to quickly find latest versions of 
the packages hosted on PyPI. We did a single request on IDE startup and showed 
outdated installed packages in the settings later. Index "/simple" however 
contains only package names and links to the dedicated pages with their 
artifacts (not for each of them, though). It means that now we have to make 
tons of individual requests to find the latest published version for each 
installed package. Isn't it going to load the service even worse?


So, yes, we're interested most in the latest version of a package. 


2016-07-13 21:57 GMT+03:00 Donald Stufft :



On Jul 13, 2016, at 2:43 PM, Dmitry Trofimov  
wrote:


Hi,


to have information about available packages, PyCharm IDE currently parses
the PyPI index page (https://pypi.python.org/pypi?%3Aaction=index).
As it is going to be deprecated soon, we are looking for a workaround.


What we need is, making one request, to get the name and the version of all 
PyPI packages. Then we cache this information in the IDE 
(https://github.com/JetBrains/intellij-community/blob/7e16c042a19767d5f548c84f88cc5edd5f9d1721/python/src/com/jetbrains/python/packaging/PyPIPackageUtil.java).


By name and version, do you mean the latest version?


—
Donald Stufft








___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig







-- 

Best regards
Mikhail Golubev___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheel including files it shouldn't

[Steve Dower]

On 27Apr2016 1445, Paul Moore wrote:

Personally, I agree with Donald that the "normal" process of building
a distribution should be:

1. Build the sdist.
2. Build wheels *from* that sdist.
3. Check the built sdist and wheels.
4. Upload the files once you've confirmed they are OK.

The tools should make that as transparent as possible (I'm not averse
to a command that builds sdist and wheels together, but such a command
could easily use the sdist to build the wheels "behind the scenes")
and there may be special cases (incremental builds of wheels when a
full recompile is a significant cost) but those are quality of
implementation details.


One extra task that I often need is the ability to separate bdist_ext 
from bdist_wheel, so that I can apply modifications to built modules 
(e.g. code signing) before packaging up the wheel.


There's probably a way to insert this step using a setuptools extension 
or a setup.py hack, but it's actually most convenient to have two 
completely separate commands (or options) for building and packaging.


Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parked Names in PyPI under user rodmena

[Steve Dower]

Thanks for the vouch, we are indeed both current Microsoft employees. I stopped 
using my work email for Python stuff when our server started corrupting URLs to 
add a phishing/malware filter.

Feel free to email the pyt...@microsoft.com address attached to the Microsoft 
user and I'll reply to you.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Donald Stufft" 
Sent: ‎4/‎19/‎2016 21:52
To: "Richard Jones" 
Cc: "disutils-sig" ; "Christopher Wilcox" 

Subject: Re: [Distutils] Parked Names in PyPI under user rodmena

I’m 100% sure Steve is a Microsoft employee and I’m like 95% sure Christopher 
is too :)


On Apr 20, 2016, at 12:24 AM, Richard Jones  wrote:


Just to be clear, are you the user "Microsoft"? You're not posting from a 
@microsoft.com email domain, is all. Or are you just a "concerned citizen"? 
Because in the case of the latter there's really nothing for me to do here 
without a request from someone actually wanting to do something with the name.




 Richard


On 20 April 2016 at 07:52, Christopher Wilcox  wrote:

DistUtils-Sig:

I was searching warehouse for all Microsoft owned packages today and came 
across a certain user that seems to have parked on a few different package 
names that I don’t believe he has any intention of using (@rodmena). 
https://warehouse.python.org/user/rodmena/ 

Can we get these released to the proper owners? He seems to have done this 
rather broadly.
If possible can the user @Microsoft be marked as the owner of the Microsoft 
package? 

Thanks!
Chris Wilcox

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig




___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig




-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Parked Names in PyPI under user rodmena

[Steve Dower]

Also the "windows" package.

Might just want to release all of those package names, as he's clearly a troll, 
but in light of the other discussions I think a case of well established and 
enforceable trademarks should be straightforward.

(Don't honestly know what we'd _do_ with packages with those names, but better 
for us to be squatting than someone else.)

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Christopher Wilcox" 
Sent: ‎4/‎19/‎2016 18:50
To: "distutils-sig@python.org" 
Subject: [Distutils] Parked Names in PyPI under user rodmena

DistUtils-Sig:

I was searching warehouse for all Microsoft owned packages today and came 
across a certain user that seems to have parked on a few different package 
names that I don’t believe he has any intention of using (@rodmena). 
https://warehouse.python.org/user/rodmena/ 

Can we get these released to the proper owners? He seems to have done this 
rather broadly.
If possible can the user @Microsoft be marked as the owner of the Microsoft 
package? 

Thanks!
Chris Wilcox___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Steve Dower]

Another alternative is making "pip the library" as has been discussed in the 
past. 

Certainly my needs would be satisfied by a library that can get me as far as 
wheel files given package name/s and version spec (and index URL I guess). 
Unpacking wheels isn't especially difficult and in my case I know there are no 
existing installs to worry about.

Top-posted from my Windows Phone

-Original Message-
From: "Paul Moore" <p.f.mo...@gmail.com>
Sent: ‎2/‎17/‎2016 2:10
To: "Robert Collins" <robe...@robertcollins.net>
Cc: "Steve Dower" <pyt...@stevedower.id.au>; "Python Distutils SIG" 
<Distutils-Sig@python.org>
Subject: Re: [Distutils] deprecating pip install --target

On 16 February 2016 at 22:52, Robert Collins <robe...@robertcollins.net> wrote:
>> An alternative would be great, though I can probably fake things somehow for
>> my purposes.
>
> Sounds similar to Daniel's need - and again, --prefix + setting PATH
> and PYTHONPATH would be better.

Note that if I read the help for --prefix correctly, "pip install
--target x foo" puts foo in x, whereas "pip install --prefix x foo"
puts foo in x/lib. So how would setting prefix allow me to put foo in
x, and not in a subdirectory? That is specifically my requirement (and
the vendoring requirement in general). I *know* that means there's no
obvious place to put data files or extensions or whatever, and that's
fine by me.

It seems that if we want to go down this route, we need to include the
full set of --install-purelib, --install-platlib, --install-scripts
etc arguments to pip. But that's probably the wrong solution - if we
want to start playing with the various install location parameters to
pip install (--target, --prefix, --root) we should probably do a
"proper" job and just find a way to allow user-defined schemes.

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] deprecating pip install --target

[Steve Dower]

I was also planning to use it in an upcoming project that has to "do its own" 
package management. The aim was to install different versions of packages in 
different directories and use sys.path modifications to resolve them at runtime 
(kind of like what setuptools did in the older days).

An alternative would be great, though I can probably fake things somehow for my 
purposes.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Vinay Sajip" 
Sent: ‎2/‎11/‎2016 15:18
To: "Distutils-Sig@Python.Org" 
Subject: Re: [Distutils] deprecating pip install --target

Robert Collins  robertcollins.net> writes:

> 
> This is fairly broken - it doesn't handle platlib vs purelib (see pip
> PR 3450), doesn't handle data files, or any other layout. Donald says
> pip uses it to maintain the _vendor subtrees only, which doesn't seem
> like a particularly strong use case.
> 
> Certainly the help description for it is misleading - since what we're
> actually doing is copying only a subset of what the package installed
> - so at a minimum we need to be much clearer about what it does.
> 
> But, I think it would be better to deprecate it and remove it... so
> I'm pinging here to see if anyone can explain a sensible use case for
> it in the context of pip :)

I use it in pretty much the same way as Paul mentioned - I wouldn't like it 
to go unless something equivalent is available. Updating the help / 
documentation for it to better reflect what it does would be uncontroversial 
for me, but I see no strong reason for deprecation and removal. As Paul 
suggests, it can get stricter about what it'll handle.

Regards,

Vinay Sajip

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] command line versus python API for build systemabstraction (was Re: build system abstraction PEP)

[Steve Dower]

As much as I dislike sniping into threads like this, my gut feeling is strongly 
pushing towards defining the Python interface in the PEP and keeping command 
line interfaces as private.

I don't have any new evidence, but pickle and binary stdio (not to mention 
TCP/HTTP for doing things remotely) are reliable cross-platform where CLIs are 
not, so you're going to have a horrible time locking down something that will 
work across multiple OS/shell combinations. There are also limits to command 
lines lengths that may be triggered when passing many long paths (if that ends 
up in there).

Might be nice to have an in-proc option for builders too, so I can handle the 
IPC in my own way. Maybe that's not useful, but with a Python interface it's 
trivial to enable.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Nathaniel Smith" 
Sent: ‎11/‎11/‎2015 4:18
To: "Robert Collins" 
Cc: "DistUtils mailing list" 
Subject: Re: [Distutils] command line versus python API for build 
systemabstraction (was Re: build system abstraction PEP)

In case it's useful to make this discussion more concrete, here's a
sketch of what the pip code for dealing with a build system defined by
a Python API might look like:

https://gist.github.com/njsmith/75818a6debbce9d7ff48

Obviously there's room to build on this to get much fancier, but
AFAICT even this minimal version is already enough to correctly handle
all the important stuff -- schema version checking, error reporting,
full args/kwargs/return values. (It does assume that we'll only use
json-serializable data structures for argument and return values, but
that seems like a good plan anyway. Pickle would probably be a bad
idea because we're crossing between two different python environments
that may have different or incompatible packages/classes available.)

-n

On Wed, Nov 11, 2015 at 1:04 AM, Nathaniel Smith  wrote:
> On Tue, Nov 10, 2015 at 11:27 PM, Robert Collins
>  wrote:
>> On 11 November 2015 at 19:49, Nick Coghlan  wrote:
>>> On 11 November 2015 at 16:19, Robert Collins  
>>> wrote:
>> ...>> pip is going to be invoking a CLI *no matter what*. Thats a hard
 requirement unless Python's very fundamental import behaviour changes.
 Slapping a Python API on things is lipstick on a pig here IMO: we're
 going to have to downgrade any richer interface; and by specifying the
 actual LCD as the interface it is then amenable to direct exploration
 by users without them having to reverse engineer an undocumented thunk
 within pip.
>>>
>>> I'm not opposed to documenting how pip talks to its worker CLI - I
>>> just share Nathan's concerns about locking that down in a PEP vs
>>> keeping *that* CLI within pip's boundary of responsibilities, and
>>> having a documented Python interface used for invoking build systems.
>>
>> I'm also very wary of something that would be an attractive nuisance.
>> I've seen nothing suggesting that a Python API would be anything but:
>>  - it won't be usable [it requires the glue to set up an isolated
>> context, which is buried in pip] in the general case
>
> This is exactly as true of a command line API -- in the general case
> it also requires the glue to set up an isolated context. People who go
> ahead and run 'flit' from their global environment instead of in the
> isolated build environment will experience exactly the same problems
> as people who go ahead and import 'flit.build_system_api' in their
> global environment, so I don't see how one is any more of an
> attractive nuisance than the other?
>
> AFAICT the main difference is that "setting up a specified Python
> context and then importing something and exploring its API" is
> literally what I do all day as a Python developer. Either way you have
> to set stuff up, and then once you do, in the Python API case you get
> stuff like tab completion, ipython introspection (? and ??), etc. for
> free.
>
>>  - no matter what we do, pip can't benefit from it beyond the
>> subprocess interface pip needs, because pip *cannot* import and use
>> the build interface
>
> Not sure what you mean by "benefit" here. At best this is an argument
> that the two options have similar capabilities, in which case I would
> argue that we should choose the one that leads to simpler and thus
> more probably bug-free specification language.
>
> But even this isn't really true -- the difference between them is that
> either way you have a subprocess API, but with a Python API, the
> subprocess interface that pip uses has the option of being improved
> incrementally over time -- including, potentially, to take further
> advantage of the underlying richness of the Python semantics. Sure,
> maybe the first release would just take all exceptions and map them
> into some text printed to stderr and a non-zero 

Re: [Distutils] warning about potential problem for wheels

[Steve Dower]

An extra data point is that we've had exactly one report of Python 3.5 not 
working due to lack of SSE, and that person was also on Windows XP (so zero 
reports on supported platforms).

That said, I should probably just fix 3.5.1 to not use SSE for core or 
distutils builds. I doubt there was a huge performance increase due to it 
(mainly in memcpy I'd assume).

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Nathaniel Smith" 
Sent: ‎10/‎10/‎2015 16:11
To: "Laura Creighton" 
Cc: "distutils-sig" 
Subject: Re: [Distutils] warning about potential problem for wheels

On Oct 10, 2015 3:37 PM, "Laura Creighton"  wrote:
>
> In a message of Sat, 10 Oct 2015 21:52:58 -, Oscar Benjamin writes:
>
> >Really this is just a case of an unsupported platform. It's unfortunate
> >that CPython doesn't properly support this hardware but I think it's
> >reasonable that if you have to build your interpreter from source then you
> >have to build your extension modules as well.
>
> Alas that there is no easy way to detect.  The situation I am
> imagining is where the administrators of a school build pythons for
> the students to run on their obsolete hardware, and then the poor
> students don't understand why pip doesn't work.  But I suppose we
> will just get to deal with that problem when and if it happens.
In case some numbers help: the numerical community has been tracking the 
deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at 
this point what I hear is:
- ~0.5% of Firefox crashes are on machines that are missing SSE2
- <0.1% of machines with Steam installed are missing SSE2
I'm not sure what python distributions like Anaconda or Activestate are doing 
wrt SSE2, but even if their builds do require SSE2 then their build tooling 
might provide a quick way to generate a whole installable environment with 
custom build options for targeting older systems. (Continuum as of today still 
hasn't released build scripts for the bottom of their stack -- 
python/numpy/etc. -- but they've claimed willingness to do so and there's 
increasing calls to make a "centos" version of Anaconda. And all the tooling 
beyond that -- e.g. the actual package manager -- is FOSS.)
-n___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Steve Dower]

Yes. SSE never makes it into the module ABI, it's just an implementation 
concern. (I believe it may be used across function/module boundaries internally 
when the compiler can prove that it's inaccessible from the outside - a more 
space effective form of inlining.)

There will be some 3.5.0-built wheels that won't work on some machines running 
3.5.1 without SSE, and some wheels will see a performance decrease when built 
with 3.5.1 vs 3.5.0 (but not compared to 3.4). These latter ones weren't 
expecting the perf increase anyway, so compatibility is likely there biggest 
concern.

This does only affect 32-bit builds, so now I'm thinking about the possibility 
of treating those as highly compatible while the 64-bit ones get better 
performance treatment, though I'm not sure how that could actually play out. It 
may help remove some of the questions about which one to use though.

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Donald Stufft" <don...@stufft.io>
Sent: ‎10/‎11/‎2015 7:31
To: "Steve Dower" <steve.do...@python.org>
Cc: "distutils-sig" <distutils-sig@python.org>; "Laura Creighton" 
<l...@openend.se>
Subject: Re: [Distutils] warning about potential problem for wheels

Will something built against 3.5.0 with SSE work on 3.5.1 without SSE? What 
about the inverse?

Sent from my iPhone

On Oct 11, 2015, at 10:17 AM, Steve Dower <steve.do...@python.org> wrote:


An extra data point is that we've had exactly one report of Python 3.5 not 
working due to lack of SSE, and that person was also on Windows XP (so zero 
reports on supported platforms).

That said, I should probably just fix 3.5.1 to not use SSE for core or 
distutils builds. I doubt there was a huge performance increase due to it 
(mainly in memcpy I'd assume).

Cheers,
Steve

Top-posted from my Windows Phone


From: Nathaniel Smith
Sent: ‎10/‎10/‎2015 16:11
To: Laura Creighton
Cc: distutils-sig
Subject: Re: [Distutils] warning about potential problem for wheels


On Oct 10, 2015 3:37 PM, "Laura Creighton" <l...@openend.se> wrote:
>
> In a message of Sat, 10 Oct 2015 21:52:58 -, Oscar Benjamin writes:
>
> >Really this is just a case of an unsupported platform. It's unfortunate
> >that CPython doesn't properly support this hardware but I think it's
> >reasonable that if you have to build your interpreter from source then you
> >have to build your extension modules as well.
>
> Alas that there is no easy way to detect.  The situation I am
> imagining is where the administrators of a school build pythons for
> the students to run on their obsolete hardware, and then the poor
> students don't understand why pip doesn't work.  But I suppose we
> will just get to deal with that problem when and if it happens.
In case some numbers help: the numerical community has been tracking the 
deployment of SSE2 to decide when to switch over for numpy/scipy builds, and at 
this point what I hear is:
- ~0.5% of Firefox crashes are on machines that are missing SSE2
- <0.1% of machines with Steam installed are missing SSE2
I'm not sure what python distributions like Anaconda or Activestate are doing 
wrt SSE2, but even if their builds do require SSE2 then their build tooling 
might provide a quick way to generate a whole installable environment with 
custom build options for targeting older systems. (Continuum as of today still 
hasn't released build scripts for the bottom of their stack -- 
python/numpy/etc. -- but they've claimed willingness to do so and there's 
increasing calls to make a "centos" version of Anaconda. And all the tooling 
beyond that -- e.g. the actual package manager -- is FOSS.)
-n
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Steve Dower]

The general rule is the minimum requirements of all supported operating 
systems, so in this case requiring SSE is a bug.

Note that this doesn't prevent us from having SSE optimized code paths, as long 
as there is a fallback to one that is more compatible. And it's also not 
binding on redistributors or package builders, who are free to add more 
restrictive requirements.

As far as identifying these sort of requirements in packages, I don't think 
they really make sense as part of a platform tag as a non-SSE build is fully 
compatible with an SSE build, but as part of version ordering it would be nice 
to be able to prefer certain tagged builds without excluding ones without the 
tag (e.g. if my platform is win32_sse2 and there's only a win32 tagged build, I 
still want it - how we formalize this is probably very complex though. Also, my 
platform tag is probably 100 chars long with everything I may want to 
represent...)

Cheers,
Steve

Top-posted from my Windows Phone

-Original Message-
From: "Oscar Benjamin" <oscar.j.benja...@gmail.com>
Sent: ‎10/‎11/‎2015 10:43
To: "Antoine Pitrou" <solip...@pitrou.net>; "Distutils-Sig@python.org" 
<Distutils-Sig@python.org>
Subject: Re: [Distutils] warning about potential problem for wheels

On Sun, 11 Oct 2015 17:44 Antoine Pitrou <solip...@pitrou.net> wrote:
On Sun, 11 Oct 2015 08:07:30 -0700
Steve Dower <steve.do...@python.org> wrote:
>
> This does only affect 32-bit builds, so now I'm thinking about the
> possibility of treating those as highly compatible while the 64-bit
> ones get better performance treatment, though I'm not sure how that
> could actually play out. It may help remove some of the questions
> about which one to use though.
That sounds reasonable to me. I don't know Windows very much, but are
there still many people using 32-bit Windows these days (on x86, I
mean)?




I don't know but I think it makes sense to follow Windows' lead. So if 3.5 
supports Vista and Vista doesn't require SSE2 then CPython shouldn't either. If 
3.6 or whatever drops support for Vista and if Windows 7 requires SSE2 then 
CPython can require it too. I assume this what happens with the OSX binaries.
Note that my recently retired computer was 64 bit and had SSE but didn't have 
SSE2 (I'm fairly sure - CPU was some budget AMD model). Also after SSE2 we have 
SSE3 etc and I've seen no indication that x86-64 manufacturers are going to 
stop adding new instructions. So this general issue isn't limited to 32 bit 
hardware and won't be solved by special casing that. I think it makes sense to 
have a general policy for architectures that will be supported by the official 
build in future.
--
Oscar___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Steve Dower]

On 14Aug2015 0038, Nathaniel Smith wrote:

Windows and OS X don't (reliably) have any package manager. So PyPI
*is* inevitably going to contain non-Python shared libraries or
statically linked modules or something like that. (And in fact it
already contains such things today.) I'm not sure what the alternative
would even be.


Windows 10 has a package manager 
(http://blogs.technet.com/b/keithmayer/archive/2014/04/16/what-s-new-in-powershell-getting-started-with-oneget-in-one-line-with-powershell-5-0.aspx) 
but I don't think it will be particularly helpful here. The Windows 
model has always been to only share system libraries and each 
application should keep its own dependencies local.


I actually like two ideas for Windows (not clear to me how well they 
apply on other platforms), both of which have been mentioned in the past:


* PyPI packages that are *very* thin wrappers around a shared library

For example, maybe libpng shows up on PyPI, and packages can then 
depend on it. It takes some care on the part of the publisher to 
maintain version-to-version compatibility (or maybe wheel/setup.py/.cfg 
grows a way to define vendored dependencies?) but this should be 
possible today.


* Packages that contain shared sources

One big problem on Windows is there's no standard place to put library 
sources, so build tools can't find them. If a package declared build 
requires libpng.x.y source then there could be tarballs somewhere (or 
even links to public version control) that have that version of the 
source, and the build tools can add the path references to include it.


I don't have numbers, but I do know that once a C compiler is available 
the next easiest problem to solve is getting and referencing sources.



Given that, the only situation I can see where we would ever
distribute wheels that require system blas on Linux, is if we were
able to do it alongside wheels that do not require system blas, and
pip were clever enough to reliably always pick the latter except in
cases where the system blas was actually present and working.


I think something similar came up back when we were discussing SSE 
support in Windows wheels. I'd love to see packages be able to run 
system checks to determine their own platform string (maybe a pip/wheel 
extension?) before selecting and downloading a wheel. I think that would 
actually solve a lot of these issues.



This means that when you build, e.g., scipy, then you get a binary
that depends on things like the in-memory layout of numpy's internal
objects. We'd like it to be the case that when we release a new
version of numpy, pip could realize hey, this new version says it has
an incompatible ABI that will break your currently installed version
of scipy -- I'd better fetch a new version of scipy as well, or at
least rebuild the same version I already have. Notice that at the
time scipy is built, it is not known which future version of numpy
will require a rebuild. There are a lot of ways this might work on
both the numpy and pip sides -- definitely fodder for a separate
thread -- but that's the basic problem.


There was discussion about an incompatible_with metadata item at one 
point. Could numpy include {incompatible_with: scipyx.y} in such a 
release? Or would that not be possible.


Cheers,
Steve


-n



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Add additional file categories for distutils, setuptools, wheel

[Steve Dower]

My brief POV is that if a package on Windows is installing anything outside 
sys.path at all then it's an application and should use something other than 
wheel for installation. WiX/MSI will do proper reference counting and upgrades 
to avoid having multiple versions colliding with each other (imagine installing 
Mercurial on 2.6 and 2.7 simultaneously...)

These should probably also bundle the interpreter as well to avoid other update 
issues. I'm already going to release a self extracting zip of the interpreter 
for this, though there will be tweaks necessary to the initialization process 
to avoid registry stomping, and tools like pynsist are being worked on to 
simply installer generation. These can freely install files anywhere, and 
having a consistent spec for generating these would be nice, but I don't want 
pip to install files outside of sys.path

Cheers,
Steve

Top-posted from my Windows Phone

From: Nick Coghlanmailto:ncogh...@gmail.com
Sent: ‎4/‎19/‎2015 9:41
To: Paul Mooremailto:p.f.mo...@gmail.com
Cc: DistUtils mailing listmailto:distutils-sig@python.org
Subject: Re: [Distutils] Add additional file categories for distutils, 
setuptools, wheel

On 19 April 2015 at 06:03, Paul Moore p.f.mo...@gmail.com wrote:
 As a possible compromise, how about an approach where on Linux system
 installs (or more accurately those install schemes that are relevant
 to the distribution packaging issue you described) the file
 categories are installed into dedicated directories, as described. But
 on other installs (virtualenvs, Windows, maybe OSX) the file
 categories map to locations within package_data, so that they can be
 accessed via normal mechanisms like loader.get_data. Application code
 would need some support code to locate and read the files, but that's
 true whether we go for this proposal or an outside of site-packages
 scheme. Also, some things may even be better designated as don't
 install in certain schemes (man files, for example, are pretty much
 useless on Windows).

That's not a compromise, that's exactly what I want to see happen :)

If it helps, one way to think of this is as a file classification
system, where the packaging tools gain the ability to sort files into
a richer set of categories, and its then up to installers to decide
how those categories map to installation locations. For Windows,
virtualenv, conda, nix, single file applications, etc, that's likely
to be self-contained application directory. For FHS, it will map to
the on-disk categorisation expected by other tools.

At the moment, because the FHS support is either hacked in, or managed
externally, there's no way for installers to remap the installation
targets for the self-contained directory use case.

Cheers,
Nick.

--
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to sign a exe created with bdist_wininst?

[Steve Dower]

It may be possible to add an empty key container to the stub with signtool so 
that it can be filled in after adding the zip without having to extend the 
length. I believe the PE header is modified to locate the certificate, so it 
doesn't necessarily have to be at the end.

Feel free to investigate this yourself with the wininst stub in 
Lib\distutils\command. I'll take a look, but may not be able to get to it for a 
while (file an issue and nosy me if you don't get anywhere, or even if you do 
and we can support this in newer versions).

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Mooremailto:p.f.mo...@gmail.com
Sent: ‎4/‎18/‎2015 2:58
To: Brian Colemailto:co...@eyesopen.com
Cc: distutils-sig@python.orgmailto:distutils-sig@python.org
Subject: Re: [Distutils] How to sign a exe created with bdist_wininst?

On 17 April 2015 at 16:17, Brian Cole co...@eyesopen.com wrote:
 We've recently converted over to using bdist_wininst for creating our
 Windows .exe installers for our libraries. Unfortunately, whenever we use
 the Windows signtool utility to cryptographically sign our installer it
 appears to corrupt the .exe and it can't be run anymore. The error message
 thrown by Windows is Setup program invalid or damaged.

 My best guess at this point is that bdist_wininst is creating a checksum of
 the file somehow and signtool is altering the file in such a way to
 invalidate that checksum. The commands we're using at this point is like
 this:

 python3.4.exe setup.py bdist_wininst --target-version 3.4 --bitmap OurLogo
 --title OurTitle-OurVersion
 cp DistUtilsSetupFileName.exe OurSetupFileName.exe
 call C:\program Files (x86)\Microsoft Visual Studio
 9.0\Common7\Tools\vsvars32.bat
 signtool sign /n OurCompany  /t
 http://timestamp.verisign.com/scripts/timstamp.dll /d OurProject /du
 OurWebsite OurSetupFileName.exe

 Anyone know of a way to cryptographically sign an .exe installer from
 bdist_wininst?

The wininst format is a stub Windows executable, with some ini-format
data and a zipfile appended (in that order). I don't know where
signtools adds the signature, but if it's at the end, then that won't
work (as it's necessary for the zip data to be the *last* thing in the
file - zipfile format supports prepending data but not appending it as
the central directory is defined as being at a fixed offset from the
end of the file).

There may also be a length or checksum in the ini data, I'd have to
check the source to confirm that. pause Just checked, no it doesn't
- the full details are here:
https://hg.python.org/cpython/file/bc1a178b3bc8/PC/bdist_wininst/install.c

So basically, I don't think it's possible to sign (or otherwise
modify) wininst executables.
Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Steve Dower]

On the Start Menu suggestion, I think that's a horrible idea. Pip is not the 
system package manager and it shouldn't be changing the system. Unversioned 
script launchers are in the same category, but aren't quite as offensive.

I know it's only a hypothetical, but I'd much rather it didn't get repeated so 
often that it actually happens. There are better tools for making app 
installers, as opposed to package installers.

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Mooremailto:p.f.mo...@gmail.com
Sent: ‎4/‎15/‎2015 17:24
To: Chris Barkermailto:chris.bar...@noaa.gov
Cc: distutils-sigmailto:distutils-sig@python.org
Subject: Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

On 15 April 2015 at 21:40, Chris Barker chris.bar...@noaa.gov wrote:
 Which brings us back to the review of extensions thing -- I think it's
 less about the end user checking it out and making a decision about it, but
 about the package builder doing that. I have a package I want to be easy to
 install on Windows -- so I go look for an extension that does the Start
 Menu, etc. Indeed, that kind of thing 'should be part of pip and/or wheel,
 but it would probably be more successful if it were done as third party
 extensions -- perhaps over the years, the ones that rise to the top of
 usefulness can become standards.

In the PEP, there's a concept of optional vs required extensions.
See https://www.python.org/dev/peps/pep-0426/#required-extension-handling.
This is crucial - I've no problem if a particular extension is used by
a project, as long as it's optional. I won't install it, so it's fine.
It seems to me that pip *has* to ignore missing optional extensions,
for this reason. Of course, that introduces the converse problem,
which is how would people who might want that extension to be
activated, know that a project used it?

Critical extensions, on the other hand, are precisely that - the
install won't run without them. I'd hope that critical extensions will
only be used for things where the installation will be useless without
them. But I worry that some people may have a more liberal definition
of required than I do. To be honest, I can't think of *anything*
that I'd consider a required extension. Console script wrappers
aren't essential, for example (you can use python -m pip even if
pip.exe isn't present). More generally, none of the extensions in PEP
459 seem essential, in this sense. Start menu entry writers wouldn't
be essential, nor would COM registration extensions necessarily be
(most of pywin32's functionality works fine if the COM components
aren't registered). Beyond that I'm struggling to think of things that
might be extensions.

So, as long as the optional vs required distinction is respected,
people are conservative about deeming something as essential, and a
missing optional extension doesn't stop an install, then I don't see
extensions as being a big issue.

Based on the above, it's possibly valid to allow required extensions
to be auto-installed. It *is* a vector for unexpected code execution,
but maybe that's OK.

Paul

PS The idea of a Start Menu entries has come up a lot here. To be
clear, I *don't* actually think such a thing is a good idea (far from
it - I think it's a pretty lousy idea) but it is a good example of
something that people think they ought to do, but in practice users
have widely differing views on whet they prefer or will use, and a
developer with limited experience could easily create a dreadful user
experience without meaning to (developer here could either mean the
extension developer, or the package developer using the extension -
both have opportunities to make a horrible mess...) So it's a good
straw man for an extension that some people will love and others will
hate :-)

PPS I'm inclined to think that the PEP should say Installation tools
MUST NOT fail if installer_must_handle is set to false for an
extension that the tool cannot process. Installation tools SHOULD NOT
attempt to install plugins or similar optional functionality to handle
an extension with installer_must_handle set to false, except with
explicit approval from the end user.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires for dev environments

[Steve Dower]

Donald Stufft wrote:
 So yea, what's the actual problem that this is attempting to solve?

ISTM (whether this is the actual intent or not) that this would be handy to 
differentiate between the dependencies needed when installing from a wheel vs. 
an sdist. Daniel's example of setup_requires including cython suggests to me 
that a wheel would include the compiled output and cython is not required in 
that case.

I don't personally have a use case for this right now, though it does seem like 
it has potential to refer to a Python package that acts as a front-end for a 
compiler (and perhaps downloader/installer... hmm...)

Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] PYD/SO platform tags (#22980)

[Steve Dower]

Hi distutils-sig,

There's a bit of discussion going on at http://bugs.python.org/issue22980 about 
extending SO tags to include bitness values, and I took the opportunity to look 
into adding platform tags for .pyd files on Windows.

There's a patch up there, but I'm interested in any thoughts or concerns people 
may have that I haven't considered. Distribution is likely to be most affected, 
and I figured distutils-sig is most likely to have people with strong opinions 
on the subject anyway, so I'm posting here :)

Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll go 
from [.pyd] to [.cp35-win32.pyd, .cp3-win32.pyd, .pyd] (where win32 
may also be win_amd64, win_ia64 or win_arm, which are the platforms that 
have #ifdefs in pyconfig.h).

By default, distutils' build_ext will use the first extension, and so builds 
will get the .cp35-win32.pyd tag. You can only get the .cp3-win32.pyd tag 
(for stable ABI) by customizing the build process - I don't think that's an 
unreasonable expectation, especially on Windows.

I can see potential for making more generic installable packages with this 
change, though I'm not pushing for that. The big win is when you're building 
and testing 32-bit and 64-bit pyds through an IDE - it greatly simplifies how 
the output directories and search paths are organized.

Thoughts/concerns?

Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PYD/SO platform tags (#22980)

[Steve Dower]

Antoine Pitrou wrote:
 Sent: Friday, December 12, 2014 1551
 To: Distutils-Sig@Python.Org
 Subject: Re: [Distutils] PYD/SO platform tags (#22980)
 
 On Fri, 12 Dec 2014 23:23:03 +
 Steve Dower steve.do...@microsoft.com wrote:
 
 Hi distutils-sig,

 There's a bit of discussion going on at http://bugs.python.org/issue22980
 about extending SO tags to include bitness values, and I took the opportunity 
 to
 look into adding platform tags for .pyd files on Windows.

 There's a patch up there, but I'm interested in any thoughts or
 concerns people may have that I haven't considered. Distribution is
 likely to be most affected, and I figured distutils-sig is most likely
 to have people with strong opinions on the subject anyway, so I'm
 posting here :)

 Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll
 go from [.pyd] to [.cp35-win32.pyd, .cp3-win32.pyd, .pyd] (where 
 win32
 may also be win_amd64, win_ia64 or win_arm, which are the platforms that
 have #ifdefs in pyconfig.h).
 
 For the record, https://www.python.org/dev/peps/pep-3149/#pep-384
 suggests abi3 as the ABI tag for the Python 3 stable ABI.

My legalistic rationale for using cp3 is that it's actually the version tag, 
not the ABI tag. It seemed from my reading that you'd get tags like 
cp35-abi3-win32, which is not helpful because you've still specified the 
minor version in the tag. Either it'll work in any CPython 3.x (cp3) or it 
needs a specific minor version (cp35), so I didn't see the value in keeping 
the ABI tag.

Of course, this only works because Windows doesn't use the ABI tag for anything 
else. Debug builds are encoded differently, and AFAICT the malloc and Unicode 
options are currently unnecessary and I'm not making any change to make them 
necessary.

I could be swayed on this point fairly easily though, if say there are other 
interpreters out there not compatible enough with CPython to use the cp tag but 
that can still use stable ABI extensions that were build assuming CPython. Or I 
could just drop the second tag and only have cp35-win32.pyd or .pyd. We've 
made it this long without a specifier, so there's no real need to go overboard 
:)

Cheers,
Steve

 Regards
 
 Antoine.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PYD/SO platform tags (#22980)

[Steve Dower]

 Donald Stufft wrote:
 On Dec 12, 2014, at 6:23 PM, Steve Dower steve.do...@microsoft.com wrote:

 Hi distutils-sig,

 There's a bit of discussion going on at http://bugs.python.org/issue22980
 about extending SO tags to include bitness values, and I took the 
 opportunity to
 look into adding platform tags for .pyd files on Windows.

 There's a patch up there, but I'm interested in any thoughts or concerns
 people may have that I haven't considered. Distribution is likely to be most
 affected, and I figured distutils-sig is most likely to have people with 
 strong
 opinions on the subject anyway, so I'm posting here :)

 Basically, importlib.machinery.EXTENSION_SUFFIXES is all that changes. It'll
 go from [.pyd] to [.cp35-win32.pyd, .cp3-win32.pyd, .pyd] (where 
 win32
 may also be win_amd64, win_ia64 or win_arm, which are the platforms 
 that
 have #ifdefs in pyconfig.h).

 By default, distutils' build_ext will use the first extension, and so builds
 will get the .cp35-win32.pyd tag. You can only get the .cp3-win32.pyd tag
 (for stable ABI) by customizing the build process - I don't think that's an
 unreasonable expectation, especially on Windows.

 I can see potential for making more generic installable packages with this
 change, though I'm not pushing for that. The big win is when you're building 
 and
 testing 32-bit and 64-bit pyds through an IDE - it greatly simplifies how the
 output directories and search paths are organized.

 Thoughts/concerns?
 
 Well here’s the code that computes what tags pip will accept:
 https://github.com/pypa/pip/blob/develop/pip/pep425tags.py
 
 Obviously any additional information put into the tag will increase the number
 of builds someone has to do in order to get full coverage, however if all of 
 the
 data in that are things that genuinely make things binary incompatible then
 that’s the right thing to do.

It could reduce the number of builds that someone has to do, as you could have 
a single wheel (for Windows) that includes all the platform-specific extensions 
rather than needing separate wheels. Whether this is an advantage or not 
deserves to be debated, but there are other installers that could benefit from 
having incompatible pyd's side-by-side in a way that Python will load the 
correct one.

Oh, it's also essential for user site-packages on Windows, since the same 
directory is used for both 32-bit and 64-bit versions of Python. That's 
actually the most compelling use-case for me, since I'm still keen to see about 
moving to a secure install for Python 3.5 and in that case we'll have more 
people using it.

Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PYD/SO platform tags (#22980)

[Steve Dower]

Steve Dower wrote:
 Antoine Pitrou wrote:
 
 My legalistic rationale for using cp3 is that it's actually the version tag, 
 not
 the ABI tag. It seemed from my reading that you'd get tags like
 cp35-abi3-win32, which is not helpful because you've still specified the 
 minor
 version in the tag. Either it'll work in any CPython 3.x (cp3) or it needs a
 specific minor version (cp35), so I didn't see the value in keeping the ABI
 tag.
 

Apologies, I got this understanding from PEP 425, not 3149/384. Using 
abi3-win32 rather than cp3-win32 would also make sense. I'm not strongly 
for or against either variant.

Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] windows overlay files

[Steve Dower]

Robin Becker wrote:
 Hi,
 I hope some windows expert can assist me with a production problem. We 
 support a
 user using windows who reports problems concerning missing attributes. Using
 GotoMeeting we inspected the file together and see that the attribute should 
 be
 present. I asked them to zip up the reportlab folder from site-packages and 
 the
 module that is in the zip does not have the attribute. Puzzlement!

Is this site-packages folder in a normal Python install in x:\Python##? Or has 
it been installed elsewhere?

 Searching reveals this
 
 Due to security features introduced with Windows Vista (UAC) any
 non-Administrator program that tries to write to protected locations
 such as Program Files will get their writes caught and redirected to
 an alternative user friendly location.

 The program that made the file will be able to see the file, but most
 other programs will not.

 Files written to protected locations will end up in a parallel file
 structure under C:\Users\[username]\AppData\Local\VirtualStore, but
 will appear to the program that created them as if actually in the
 intended location.
 
 
 so I'm wondering if this is such an issue. The user has a 4 year old version 
 of
 reportlab and doesn't wish to upgrade. In the past, when they had XP, we used 
 to
 support minor fixes by modifying the modules and having them overwrite the
 installed versions in site-packages files.

How does the overwrite work? If they are copying files in Explorer, the 
redirection you quote above should not occur, but if you have a separate 
program to do the copying then it will if the program is not run as 
administrator.

 Clearly if some kind of security issues are in place which causes this kind of
 double file problem, then overwriting the python will not always work. In
 addition it may be that users can't write the pyc or something.

The pyc issue should be resolved by generating them at install time. There's an 
option in the Python installer for this, but any packages added later will need 
to do it themselves (I believe all of the standard installers do it - 
pip/easy_install/bdist_win).

 Does anyone here have experience of these issues? Will I be forced to maintain
 patched installers etc etc? Is there some trick like having an administrator
 write the files?

If the files need to be written into Program Files, then you'll need to be an 
administrator - you're making a machine-wide change, which regular users don't 
have permission to do.

That's about all the info I can provide without more details about the 
customer's file system layout (full paths are really necessary for figuring 
this out) and the programs/processing being used to install and update these 
files.

Cheers,
Steve

 --
 Robin Becker

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows?

[Steve Dower]

Ben Finney wrote:
 Steve Dower steve.do...@microsoft.com writes:
 Ben Finney wrote:
  The restrictions of the license terms make MS Windows an
  unacceptable risk on any machine I'm responsible for.

 Just out of interest, which restrictions would those be?
 
 It has been a long time since I bothered to read any of the numerous license
 texts from Microsoft, so I can't cite specific clauses. From memory,
 unacceptable restrictions include:
 
 * Restricting the instance to specific hardware, instead of leaving it
 up to the recipient to run the work they paid for on any hardware they
 choose.

If by specific hardware you mean the one-license-per-user-per-machine rule, 
you probably want to consider Windows Server, which has a more flexible license 
in this respect (or maybe not - it might just allow multiple users on one 
license/machine. I haven't checked this).

 * Forbidding reverse-engineering of the OS to see how it behaves.

Yeah, I doubt that restriction is moving anywhere. It's standard for 
closed-source software, and as I understand it's intended to legally protect 
trade secrets and patents (i.e. we tried our hardest to keep this a trade 
secret). I've never heard of anyone being pursued for doing it though, except 
to be offered a job working on Windows :)

 * Forbidding collaboration with other recipients to discover how the OS
 behaves.

Other recipients are explicitly excluded - for use by one person at a 
time[1] - so the rest of this point doesn't really make any sense to me.

That said, it does trigger some memories of when I was contributing to ReactOS 
years ago... is this one of their suggestions about how to avoid taint? (Or 
maybe from Wine?) Those guys have obtained their own legal advice which is 
going to be aimed at preventing a court case (not just preventing a loss - 
preventing it from happening in the first place) and so it's going to be based 
on an interpretation of the license and be more defensive than most people need 
to worry about.

 * Refusal to disclose the source code for the running OS to the
 recipient.

Again, it's part of the business and legal model. If you really want access to 
the source code, you can pay for it, but most people and businesses can't 
afford it or don't want it that badly. (There are also technical reasons why 
the source code can't easily be disclosed - how many hundreds of gigabytes of 
code are you willing to download and wade through? Yes, it's that big.)

 * Forbidding the recipient from getting their choice of vendor to make
 improvements to the OS and collaborate with other recipients on the
 improvements.

I know this used to exist, as there were a number of RT/embedded OSs available 
that were based on Windows. I think at this point they've all been absorbed 
into Microsoft though.

 * Arrogating control of the running OS to a party other than the license
 recipient, including the ability to (at Microsoft's sole discretion)
 deny applications to run, and to disable features of the OS.
 
 * Arrogating data collection to Microsoft and undisclosed third parties,
 tracking broad classes of activity on the OS and sending the logs to a
 server not of the recipient's choosing.

It seems you fundamentally disagree with the 'licensing' model and would prefer 
an 'ownership' model. That's fine, but it's not the business model Windows 
operates under and that is unlikely to ever change. Even if I were CEO, I'd 
have a hard time changing that one :)

 Does this prevent you from creating a VM on a cloud provider on your
 own account?
 
 If I need to accept restrictions such as the above, I don't see that the
 location of the instance (nor the fees charged) has any affect on these
 concerns. The risks discussed above are not mitigated.
 
 If the licensing is a real issue, I'm in a position where I can have a
 positive impact on fixing it, so any info you can provide me (on- or
 off-list) about your concerns is valuable.
 
 Thank you for this offer, I am glad to see willingness expressed to solve 
 these
 restrictions. I hope you can achieve software freedom for all recipients of
 Microsoft operating systems.
 
 Until then, the risk is too great to anyone to whom I have professional
 responsibilities, and my advice must continue to be that they avoid accepting
 such restrictions.

That's a fair enough position, and without people taking that stance, Linux 
(and practically every OS that's based on it) wouldn't be anywhere near as 
usable as it is today. I'm also fully aware of people with the exact opposite 
stance who give the exact opposite advice, so there's room in this world for 
all of us.

I'm sorry I can't do any better than the few responses above - these are big 
issues that run to the core of how Microsoft does business, and not only am I 
incapable of changing them, I'm nowhere near capable of fully understanding how 
it all fits together. Thanks for being willing to engage, though. It's always 
valuable to hear alternative points

Re: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows?

[Steve Dower]

True. It is followed immediately by a clarification that local laws take 
precedence, and I guess it's still sufficient to cover the trade secret side of 
things. Licenses that have to apply internationally are tricky :)

Top-posted from my Windows Phone

From: Jan Claeysmailto:li...@janc.be
Sent: ‎11/‎10/‎2014 17:33
To: distutils-sig@python.orgmailto:distutils-sig@python.org
Subject: Re: [Distutils] Call for information - What assumptions can I make 
about Unix users' access to Windows?

Steve Dower schreef op ma 10-11-2014 om 16:35 [+]:
  * Forbidding reverse-engineering of the OS to see how it behaves.

 Yeah, I doubt that restriction is moving anywhere. It's standard for
 closed-source software, and as I understand it's intended to legally
 protect trade secrets and patents (i.e. we tried our hardest to keep
 this a trade secret). I've never heard of anyone being pursued for
 doing it though, except to be offered a job working on Windows :)

FWIW: that statement is illegal and thus void in e.g. the EU (and I
thought even the USA?).  That's probably why it didn't get pursued often
recently...  ;)


--
Jan Claeys

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Call for information - What assumptions can I make about Unix users' access to Windows?

[Steve Dower]

Ben Finney wrote:
 Paul Moore p.f.mo...@gmail.com writes:
 
 To that end, I'd like to get an idea of what sort of access to Windows
 a typical Unix developer would have. […] Ideally, a clean Windows 7 or
 later virtual machine is the best environment, but I don't know if
 it's reasonable to assume that.
 
 It's difficult to say what “a typical Unix developer” is. But a significant 
 use
 case is going to be “no legal access to any MS Windows instance”.
 
 The restrictions of the license terms make MS Windows an unacceptable risk on
 any machine I'm responsible for.

Just out of interest, which restrictions would those be? I may be able to raise 
them with one of our lawyers and get some clarification.

 It has been many years since I've even had a colleague who has a MS Windows
 instance, and I am not sure where I'd go for one if the need arose.

 If I was required to provide packages for MS Windows, the only viable 
 solutions
 would be those that don't involve me obtaining an MS Windows instance myself.

Does this prevent you from creating a VM on a cloud provider on your own 
account? As far as Microsoft Azure is concerned, this is well within the 
license restrictions (at least for Windows Server right now), and all providers 
giving you access to Windows should be bundling in a license fee, which makes 
it about as legit as possible. Simply giving you share time on someone else's 
copy of Windows is much more of a grey area as far as licensing is concerned.

If the licensing is a real issue, I'm in a position where I can have a positive 
impact on fixing it, so any info you can provide me (on- or off-list) about 
your concerns is valuable.

Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[Steve Dower]

David Genest wrote:
 1) add the dependent dlls to every package that needs it (Steve's answer
 https://mail.python.org/pipermail/distutils-sig/2014-September/024982.html
 concurs that the dependent dll would be loaded only once)

This is the best approach regardless of what else works/doesn't work. Doing 
this will ensure that your extension module loads the correct dependencies 
regardless of what other packages or versions are installed in the same 
environment.

The Scripts folder is not necessarily on the path for all users, and nor is the 
Python directory itself. Depending on how you launch Python, you may get DLLs 
loaded from the Python directory or the Scripts directory, but either way you 
should _always_ get DLLs loaded from the same directory as the extension 
module. Neither directory is intended to be a dumping ground for all the 
dependencies used by packages.

I see no reason 'pip wheel' should produce different wheels to bdist_wheel (but 
perhaps there is one?), so I would consider this a bug.

Cheers,
Steve

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7

[Steve Dower]

Olivier Grisel wrote:
 Thank you very Steve for pushing that installer out, this is very appreciated.
 
 What is the story for project maintainers who want to also support Python 3.3+
 (for 32 bit and 64 bit python) for their project with binary wheels for 
 windows?
 At the moment it's possible to use the Windows SDK as documented here:
 
 http://scikit-learn.org/dev/install.html#building-on-windows
 
 However getting VC Express + Windows SDK is hard and slow to setup and cannot 
 be
 scripted in a CI environment.

It can be, but there are a few tricks involved...
 
 In the mean time, it's possible to use CI environments that already feature 
 all
 the necessary versions of the VC compilers and libraries such as appveyor.com,
 see this demo project:
 
 https://github.com/ogrisel/python-appveyor-demo
 https://ci.appveyor.com/project/ogrisel/python-appveyor-demo

This is the best way to have it set up - create a base VM image for your CI 
environment manually and clone it. I believe all the major cloud providers 
support this, though using a CI specialist like Appveyor makes it even easier.

As far as the future story, it will probably be move to 3.5 on VC14 as soon as 
possible. Internally, I'll be pushing for a CI-compatible installer for our 
build tools, which I expect will actually get quite a bit of traction right now.

Unfortunately, going back in time to do it for both VC9 and VC10 was not an 
option. We chose VC9 because 2.7 is where people are stuck, while migrating 
from 3.3-3.5 should not be as big an issue.

Cheers,
Steve

 --
 Olivier
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Wheels and dependent third party dlls on windows

[Steve Dower]

David Genest wrote:
 Subject: Re: [Distutils] Wheels and dependent third party dlls on windows
 
 
 This is not true. Python loads DLLs with
 LOAD_WITH_ALTERED_SEARCH_PATH, to allow them to be located alongside the pyd
 file. You should therefore be able to ship the dependent dll in the package
 directory (which wheels support fine).
 
 Paul
 
 Ok, so what if the dll is shared in a given environment (multiple extensions 
 use
 it)?, the shared dll should be copied to every package? Won't that cause
 multiple loads by the system?

A DLL can only be loaded once per process (python.exe, in this case) and it 
will be loaded based on its file name (not the full path). Whoever loads first 
will win every future load for the same filename.

If you're loading it directly, it's fairly easy to rename a DLL to something 
likely to be unique to your project (or at least to put a version number in it) 
so that other packages won't use it. There are more complicated approaches 
using manifests and activation contexts (this is how different .pyd files with 
the same name can be correctly loaded), but ensuring a unique name is much 
easier.

If the DLL is loaded implicitly by a .pyd, then as Paul says it should be 
loaded correctly if it is alongside the .pyd.

Dependency Walker from www.dependencywalker.com is a great tool for checking 
what DLLs will be loaded by an executable or DLL. I recommend enabling 
profiling of your python.exe process when you try and import your packages to 
see where it is looking for its dependencies.

Hope that helps,
Steve

 Thanks for your response,
 
 D.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7

[Steve Dower]

It's free (VC Express 2008 is behind a pay wall these days)
It's small (85MB download, 300mb on installed)
It's a per-user install with no reboot required

If you have the permissions, time, and access for VC Express 2008, it gains you 
nothing. You're not the intended target audience (I thought I had that wording 
in the announcement, but I guess not).

Most people don't have or want Visual Studio installed on their machine, or 
need to install on a machine where they're not admin (think university student 
on a lab machine who needs Cython).

Cheers,
Steve

Top-posted from my Windows Phone

From: Piotr Dobrogostmailto:p...@2014.dobrogost.net
Sent: ‎9/‎27/‎2014 3:34
To: Steve Dowermailto:steve.do...@microsoft.com
Cc: distutils sigmailto:distutils-sig@python.org
Subject: Re: [Distutils] Microsoft Visual C++ Compiler for Python 2.7


On Sep 27, 2014 12:32 AM, Steve Dower 
steve.do...@microsoft.commailto:steve.do...@microsoft.com wrote:

 I'll post this on the various other lists later, but I promised distutils-sig 
 first taste, especially since the discussion has been raging for a few days 
 (if you're following the setuptools repo, you may already know, but let me 
 take the podium for a few minutes anyway :) )

 Microsoft has released a compiler package for Python 2.7 to make it easier 
 for people to build and distribute their C extension modules on Windows. The 
 Microsoft Visual C++ Compiler for Python 2.7 (a.k.a. VC9) is available from: 
 http://aka.ms/vcpython27

 This package contains all the tools and headers required to build C extension 
 modules for Python 2.7 32-bit and 64-bit (note that some extension modules 
 require 3rd party dependencies such as OpenSSL or libxml2 that are not 
 included). Other versions of Python built with Visual C++ 2008 are also 
 supported, so Python 2.7 is just advertising - it'll work fine with 2.6 and 
 3.2

What that buys us in comparision to simply using VC 2008 Express?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Microsoft Visual C++ Compiler for Python 2.7

[Steve Dower]

I'll post this on the various other lists later, but I promised distutils-sig 
first taste, especially since the discussion has been raging for a few days (if 
you're following the setuptools repo, you may already know, but let me take the 
podium for a few minutes anyway :) )

Microsoft has released a compiler package for Python 2.7 to make it easier for 
people to build and distribute their C extension modules on Windows. The 
Microsoft Visual C++ Compiler for Python 2.7 (a.k.a. VC9) is available from: 
http://aka.ms/vcpython27 

This package contains all the tools and headers required to build C extension 
modules for Python 2.7 32-bit and 64-bit (note that some extension modules 
require 3rd party dependencies such as OpenSSL or libxml2 that are not 
included). Other versions of Python built with Visual C++ 2008 are also 
supported, so Python 2.7 is just advertising - it'll work fine with 2.6 and 
3.2.

You can install the package without requiring administrative privileges and, 
with the latest version of setuptools (from the source repo - there's no 
release yet), use tools such as pip, wheel, or a setup.py file to produce 
binaries on Windows.

The license prevents redistribution of the package itself (obviously you can do 
what you like with the binaries you produce) and IANAL but there should be no 
restriction on using this package on automated build systems under the usual 
one-developer rule (http://stackoverflow.com/a/779631/891 - in effect, the 
compilers are licensed to one user who happens to be using it on a remote 
machine).

My plan is to keep the download link stable so that automated scripts can 
reference and install the package. I have no idea how long that will last... :)

Our intent is to heavily focus on people using this package to produce wheels 
rather than trying to get this onto every user machine. Binary distribution is 
the way Windows has always worked and we want to encourage that, though we do 
also want people to be able to unblock themselves with these compilers.

I should also point out that VC9 is no longer supported by Microsoft. This 
means there won't be any improvements or bug fixes coming, and there's no 
official support offered. Feel free to contact me directly 
steve.do...@microsoft.com if there are issues with the package.

Cheers,
Steve

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers

[Steve Dower]

Chris Barker wrote:
 On Wed, Sep 24, 2014 at 6:55 AM, Paul Moore p.f.mo...@gmail.com wrote:
 Thanks for the pointer. (Also thanks to Allen Riddell). I'll take a
 look. Ideally, what I'd like to do is write something up to help
 non-Windows experts get things up and running, so this will be very
 useful.
 
 Thanks -- that would be great. But really, why is this so hard? Win64 is
 essentially One platform, and the freely available SDK is ONE compiler
 environment.
 
 surely it's possible to write a batch script of some sort that you could put
 somewhere (or even deliver with python! ) so this would be:
 
 1) download and install THIS (the sdk from MS)
 
 2) run:
 set_up_win_complier.py
 
 3) build the package:
 python setup.py build
 
 without needing to do multiple step, without needing to be in the special 
 set-up
 command Window, etc.
 
 In fact, even better would be for distutils to run the mythical
 set_up_win_complier.py script for you.
 
 distutils does work out of the box with the VS2008 Express for 32 bit -- I'm
 still confused why this is so much harder for 64 bit.

Someone made a decision back when that express edition was released that people 
who _needed_ 64-bit compilers could justify paying for them. At the time 
(pre-Windows 7, which was the first usable 64-bit Windows), this made sense, 
but the world has changed since then and so have the later versions of VC++ 
Express/Express for Desktop, which now include all the compilers.

 *sigh*

As I mentioned at the start of this thread - hold your frustration and wait for 
a little while :)

Cheers,
Steve

 -Chris
  

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Building Python extensions on 64-bit Windows using the SDK compilers

[Steve Dower]

We're very close to having some good news, but unfortunately, that's all I can 
say right now. Expect a more significant email/announcement from me in the next 
couple of weeks. (Distutils will hear it first and get the most detailed info.)

Sent from my Windows Phone

From: Paul Mooremailto:p.f.mo...@gmail.com
Sent: ‎9/‎23/‎2014 15:42
To: Distutilsmailto:distutils-sig@python.org; Steve 
Dowermailto:steve.do...@microsoft.com
Subject: Building Python extensions on 64-bit Windows using the SDK compilers

Can anyone give me some advice, please? I am trying to build
extensions on Windows 64-bit, using the free Windows SDK compilers.
But I can't find any official documentation on how to do this, and
everything I have tried so far has failed in frustrating ways. I'm now
at the point where I appear to be hitting the following bug -
http://bugs.python.org/issue7511 which has stumped me completely.
Sadly, as is typical with distutils issues, this one seems to have
been round for years and there is little or no sign that anyone is
willing to fix it.

Two questions, really:

* Is there any intention that building extensions with the SDK
compilers is supported?
* How do I do it, if so?

Personally, this is of limited relevance, as I have the full version
of MSVC available. But I'm trying to put together some documentation
for package developers on how to build Windows wheels, in particular
using Appveyor to automate the process, with the intention that people
shouldn't have to jump through hoops to provide wheels, but should
rather be able to simply use a prebuilt recipe to automate the
process.

As an alternative, I wonder whether Microsoft would be willing to
support Appveyor by providing them with access to the full version of
MSVC (2008 and 2010) for the build workers? Steve - do you know if
there's any possibility of something like that?

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Create formal process for claiming 'abandoned' packages

[Steve Dower]

Donald Stufft wrote:
 Perhaps in Warehouse the procedure can be automated to some degree
 and a public record of what actions were taken and when? I don’t mean like 
 a public log of the actual email address or email content or anything of the
 sort. Just like a attempted to contact on X date, notified X thing on Y,
 No response in X time, transfering ownership kind of things.

 Maybe we could create something like python-updates which would be a read only
 mailing list which just posts a thread per request and updates it with the
 actions taken and stuff. People who care could subscribe to it without having
 to get all of distutils-sig or wahtever.

 Maybe it could even offer package authors the ability to mark a package as
 Request For Adoption saying that they have a package that they wrote, but
 that they no longer wish to maintain.
 
 I don't know, I'm just tossing out some potentional ideas!

In the same vein, but at the more annoying end of the scale (I don't know how 
big a problem this really is) - build in an expiry date for every package. 
Every upload or ping from the maintainer extends it by 6-12 months, then 
automate a couple of emails in the last month before it expires. Nothing more 
has to happen, except that then it may be claimed by someone else and the 
communication has already been done. The manual process becomes checking a flag 
and the rest is automated. Regular emails also helps make sure that you track 
email address changes better.

I also like the Request for Adoption idea. Plenty of projects get wide use 
without developing a community where it's easy to find someone to take over.

Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Other ideas from today's packaging meetup at EuroPython

[Steve Dower]

Will that also allow me to put ‘macports’ or ‘homebrew’ in there you can 
create an Cython-0.20.1-cp27-none-lmacosx_x86_64-macports_10.10.whl 
distribution?

Just how many wheels are people going to have to publish? Who has that many dev 
machines? Without a build farm, I can't see this being more helpful than 
frustrating (and the build farm is going to need a way to automatically get 
non-Python dependencies, which is not our business, so that's a long way off 
IMO).

An organisation building their own wheels in-house and configuring their 
machines to use an OS specific index/wheelhouse sounds like a much more 
feasible idea that works now and could do with more traction and encouragement. 
(Someone mentioned earlier that they do this - apologies for forgetting who.)

Cheers,
Steve

Top-posted from my Windows Phone

From: Wichert Akkermanmailto:wich...@wiggy.net
Sent: ‎7/‎25/‎2014 22:27
To: Nick Coghlanmailto:ncogh...@gmail.com
Cc: DistUtils mailing listmailto:distutils-sig@python.org
Subject: Re: [Distutils] Other ideas from today's packaging meetup at EuroPython


On 26 Jul 2014, at 00:08, Nick Coghlan 
ncogh...@gmail.commailto:ncogh...@gmail.com wrote:


On 26 Jul 2014 05:56, Donald Stufft 
don...@stufft.iomailto:don...@stufft.io wrote:

 On July 25, 2014 at 3:50:30 PM, Wichert Akkerman 
 (wich...@wiggy.netmailto:wich...@wiggy.net) wrote:
 Will that guarantee the OS-provided Python was used? Or is there still a 
 risk someone was using a custom compiled Python on an Ubuntu 14.04 system 
 that is not binary compatible with the Ubuntu-provided Python?

 Wichert.


 No It won’t guarantee the OS-provided Python is used. It doesn’t even 
 guarantee that the OS provided libs are being linked to. However at that 
 point you’ve more or less reached parity with Windows and OSX where Wheels 
 (and Eggs before them) are generally built to target the “System” Python and 
 if you’re not using the “System” Python you might end up having a bad time.

As Donald says, we haven't worked out the exact technical details yet, but the 
general idea is that just as a Windows or Mac OS X wheel published on PyPI 
should be binary compatible with the corresponding 
python.orghttp://python.org/ installer, it should be possible to get the 
relevant info into wheel filenames for people to be able to indicate the 
expected contents of /etc/os-release.

I suspect that for Linux you mean “system-provided Python”? Looking at 
https://www.python.org/downloads/release/python-341/ there is no 
python.orghttp://python.org binary installer for Linux. Even if there was I 
would expect only a small number of people to use that over the system-provided 
version, considering Linux distributions generally do a excellent job packaging 
Python.

An /etc/os-release approach does sound like a reasonable approach. Will that 
also allow me to put ‘macports’ or ‘homebrew’ in there you can create an 
Cython-0.20.1-cp27-none-lmacosx_x86_64-macports_10.10.whl distribution? That 
might be useful. It would be slightly abusing the spec though since macports 
does not have releases itself so we’re mixing an OSX version number in, but 
that does not feel any worse than the other binary compatibility guarantees.

For Linux why not use the existing /etc/lsb_release instead of creating a new 
file?

Another way to go, if softwarecollections.orghttp://softwarecollections.org/ 
(aka virtualenv for your distro package manager) gets any traction whatsoever 
on the Debian/Ubuntu side of the world, would be to publish wheels based on 
*software collection* compatibility, rather than distro compatibility.

I had never heard of that; I’ll need to take a look at it. A quick look at the 
website doesn’t really tell me what problem it is trying to solve or how it 
works. It might be in the same space as things like debootstrap, docke or 
PPSar. Or perhaps NixOS really is a better way to handle this.


That would be appealing to me wearing both my packaging hats - making prebuilt 
binaries for libraries more readily available to Python users in a way that 
maps to Python versions rather than distro variants, while at the same time 
encouraging them to get their end user applications the heck out of the system 
Python so they aren't tied so closely to the distro upgrade cycle :)

Isn’t that already handle in the Debian/Ubuntu world with PPAs? Those do make 
it completely trivial to install extra things on a system, and you could easily 
make PPAs that provide custom builds that don’t overlap the OS.

Wichert.

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] pip, virtualenv, setuptools on Windows

[Steve Dower]

Paul Moore wrote:
 On 7 July 2014 15:25, M.-A. Lemburg m...@egenix.com wrote:
 Yes: Installers for Python 2.6, 2.7 and 3.3 :-)

 Ha. That one, I'll leave to someone who cares about 2.x... ;-)

 Paul

That someone is Nick Coghlan, and as long as the python-dev discussion 
doesn't take too long, it should happen for 2.7.x (where x may be 10+, at the 
rate OpenSSL bugs keep forcing us to rerelease...)

There's also https://bootstrap.pypa.io/get-pip.py, which is better than an MSI 
in almost every way IMHO.

Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Windows 8 pkg_resources permissions bug

[Steve Dower]

So it isn't a trivial repro, which means there is some more context required 
from anyone who sees this issue.

I personally would be looking at the ACLs of the files that can't be accessed. 
I expect to see that they are not inheriting permissions from their parent and 
probably only have read-only permissions for whoever installed - this is easy 
to check, because the user won't be able to open the file in any other program 
either (or look at the permissions for that matter...). The admin user will be 
able to see them and restore them.

The logic in setuptools/pip/distutils/setup.py/whoever that may lead to this 
is these files should _really_ be read-only and so they reset all permissions 
and give the current user read access. If the administrative user is the 
current user (which is pretty normal for Windows these days, because even the 
admins spend most of their time restricted), this is fine for a single-user 
system. However, where the admin user is a separate account (name/SID) from the 
interactive user, this will result in the normal user being denied even read 
access.

My testing (with pip install flake8 in Python 2.7.6 64-bit) didn't show anyone 
changing the permissions on any files. However, it is possible that the user 
has manually secured their Python install (else why would they be running pip 
install as admin?). AFAICT, Python 2.7 doesn't provide a way in the stdlib to 
change ACLs (os.chmod is totally restricted to the read-only flag, which isn't 
the problem here), and I didn't see anything in pip or setuptools that could do 
it (though setuptools will use pywin32 if it is there - it didn't seem related, 
but I also didn't test with it installed).

My specific questions for the user would be:
* Is your administrator account a different account from the one you normally 
log in with? (Alternatively, do you see the UAC dialog appear? Do you have to 
type a password or do you just click Yes?)
* Can you open the file in Notepad or equivalent?
* If you run cacls filename as administrator, what is the output?
* If you run cacls C:\Python27\python.exe, what is the output?

This sort of case is the best argument for secure-by-default installations into 
Program Files - no installer should even have to think about messing with ACLs. 
The current CPython installers support this sort of install if you change the 
directory, but maybe it is time for pip/setuptools to start assuming that user 
site-packages is the default on Windows so we could actually consider changing 
the default Python install location? (If anyone really wants to dive into that 
discussion again, go ahead and change the subject line :) )

Cheers,
Steve 

Steve Dower wrote:
 My guess would be that something is setting permissions for the current user,
 and these installs were done by an admin user who is not the current user. 
 This
 will result in explicit permissions on the file that may deny read access.

 I'll have to confirm tomorrow. Do you have the exact versions of setuptools
 and pip that are problematic? (My guess is that it won't matter. The code has
 probably been in there for a while if I'm right.)
 
 Cheers,
 Steve
 
 From: Ian Cordasco
 Sent: ‎5/‎5/‎2014 17:27
 To: Distutils list
 Subject: [Distutils] Windows 8 pkg_resources permissions bug
 Hey all,
 
 I searched the issues on the setuptools repository but I couldn't find
 any that seemed relevant to what a user of Flake8 has run into:
 https://bitbucket.org/tarek/flake8/issue/142/installation-on-windows-8-permissions
 
 It seems that installing something that relies on pkg_resources to
 load plugins will cause errors on Windows 8. Does anyone have any
 insight? Is this actually a bug? Are we just using pkg_resources
 incorrectly?
 
 Thanks in advance,
 Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Windows 8 pkg_resources permissions bug

[Steve Dower]

My guess would be that something is setting permissions for the current user, 
and these installs were done by an admin user who is not the current user. This 
will result in explicit permissions on the file that may deny read access.

I'll have to confirm tomorrow. Do you have the exact versions of setuptools and 
pip that are problematic? (My guess is that it won't matter. The code has 
probably been in there for a while if I'm right.)

Cheers,
Steve

To-posted from my Windows Phone

From: Ian Cordascomailto:graffatcolmin...@gmail.com
Sent: ‎5/‎5/‎2014 17:27
To: Distutils listmailto:distutils-sig@python.org
Subject: [Distutils] Windows 8 pkg_resources permissions bug

Hey all,

I searched the issues on the setuptools repository but I couldn't find
any that seemed relevant to what a user of Flake8 has run into:
https://bitbucket.org/tarek/flake8/issue/142/installation-on-windows-8-permissions

It seems that installing something that relies on pkg_resources to
load plugins will cause errors on Windows 8. Does anyone have any
insight? Is this actually a bug? Are we just using pkg_resources
incorrectly?

Thanks in advance,
Ian
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] ensurepip in Python 2.7

[Steve Dower]

And just as I post that, I figure it out. Guess I will be doing the next build.

Cheers,
Steve

Top-posted from my Windows Phone

From: Steve Dowermailto:steve.do...@microsoft.com
Sent: ‎5/‎3/‎2014 7:48
To: Nick Coghlanmailto:ncogh...@gmail.com; DistUtils mailing list 
mailto:distutils-sig@python.org
Subject: Re: [Distutils] ensurepip in Python 2.7

If I can get everything to build, that is... :-\ (it's not too bad - just the 
64-bit installer giving me trouble, and Martin's helping out. 2.7.7's installer 
may be late...)

On the bright side, Microsoft has apparently decided that building Python 
installers for Windows makes good business sense, so they're fully supportive. 
(Or maybe they just know it's a good way to keep me there for longer ;) )

Cheers,
Steve

Top-posted from my Windows Phone

From: Nick Coghlanmailto:ncogh...@gmail.com
Sent: ‎5/‎2/‎2014 22:01
To: DistUtils mailing list mailto:distutils-sig@python.org
Subject: [Distutils] ensurepip in Python 2.7


Hey all,

The next 2.7 release is coming up in a few weeks, and work is still needed on 
getting the SSL infrastructure updated.

That release will likely also be the first one with Steve Dower at the helm for 
the creation of the Windows installers.

Accordingly, I think it makes sense to leave proposing backporting ensurepip 
until 2.7.8 in November at the earliest. (Fortunately, there's no pyvenv in 
2.7, so unbundling for the Linux distros is much simpler)

Cheers,
Nick.
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distribution and packaging mini-summit at PyCon: today 5 to 6

[Steve Dower]

I'll be there too.

Top-posted from my Windows Phone

From: Daniel Holthmailto:dho...@gmail.com
Sent: ‎4/‎11/‎2014 13:22
To: Éric Araujomailto:mer...@netwok.org
Cc: DistUtils mailing list mailto:distutils-sig@python.org
Subject: Re: [Distutils] Distribution and packaging mini-summit at PyCon: today 
5 to 6


See you there

On Apr 11, 2014 1:21 PM, Éric Araujo 
mer...@netwok.orgmailto:mer...@netwok.org wrote:
  Hi all,

  Jason is leaving tomorrow so Nick and I figured the best time to meet
and talk about the Next Packaging Things would be today from 5pm to 6pm,
before the PyCon Dinners.

  I’m going to run to the 5th floor now and book one of the rooms.

  Cheers
___
Distutils-SIG maillist  -  
Distutils-SIG@python.orgmailto:Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distribution and packaging mini-summit at PyCon: today 5 to 6

[Steve Dower]

I've also let the conda guys know, so we'll have someone from their team too 
(probably Travis).

From: Steve Dowermailto:steve.do...@microsoft.com
Sent: ‎4/‎11/‎2014 14:57
To: Daniel Holthmailto:dho...@gmail.com; Éric Araujomailto:mer...@netwok.org
Cc: DistUtils mailing listmailto:distutils-sig@python.org
Subject: Re: [Distutils] Distribution and packaging mini-summit at PyCon: today 
5 to 6

I'll be there too.

Top-posted from my Windows Phone

From: Daniel Holthmailto:dho...@gmail.com
Sent: ‎4/‎11/‎2014 13:22
To: Éric Araujomailto:mer...@netwok.org
Cc: DistUtils mailing list mailto:distutils-sig@python.org
Subject: Re: [Distutils] Distribution and packaging mini-summit at PyCon: today 
5 to 6


See you there

On Apr 11, 2014 1:21 PM, Éric Araujo 
mer...@netwok.orgmailto:mer...@netwok.org wrote:
  Hi all,

  Jason is leaving tomorrow so Nick and I figured the best time to meet
and talk about the Next Packaging Things would be today from 5pm to 6pm,
before the PyCon Dinners.

  I’m going to run to the 5th floor now and book one of the rooms.

  Cheers
___
Distutils-SIG maillist  -  
Distutils-SIG@python.orgmailto:Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] distlib.mount API design (was: wheels on sys.path clarification (reboot))

[Steve Dower]

FWIW, Windows (by default) has a regular maintenance task that will clean up 
old files in the TEMP directory. I think the default settings will delete files 
older than 30 days and more aggressively if disk space is running low.

I'd say pick a consistent/static subfolder ('wheel_mount_35_amd64' or 
something), autogenerate whatever is needed within there, and leave it behind. 
Users who are concerned can rm -rf whenever they like and everyone else can let 
the OS handle it.

Cheers,
Steve

Top-posted from my Windows Phone

From: Paul Mooremailto:p.f.mo...@gmail.com
Sent: ‎1/‎30/‎2014 14:52
To: Nick Coghlanmailto:ncogh...@gmail.com
Cc: DistUtils mailing listmailto:distutils-sig@python.org
Subject: Re: [Distutils] distlib.mount API design (was: wheels on sys.path 
clarification (reboot))

On 30 January 2014 22:38, Nick Coghlan ncogh...@gmail.com wrote:
 The advantage of wheels over plain zipfiles for this use case is the
 structured metadata. distlib.mount doesn't try to guess the package
 structure for the extensions, you have to provide an EXTENSIONS file in the
 metadata that explains what C extensions are present and how they should map
 into the module namespace.

OK, I think I get the idea now.

I'm still not comfortable with the temp directory clutter that
unpacking leaves (in particular on Windows where deletion isn't even
possible in an atexit routine) but I'll survive.

I *would* like to see the various technical issues and implications in
the API documentation, though. The implications and limitations, and
in particular the manual cache management requirements, need to be
made explicit. (I thought I'd seen docs somewhere, but they definitely
aren't in the API reference for the distlib.wheel module).

Paul
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Test Windows Installers

[Steve Dower]

I've done basic testing (install, pip install, pip list, pip uninstall, repair, 
uninstall) against:

WinXP SP3 x86
Vista SP2 x86/x64
Win7 SP1 x86/x64
Win8 RTM x86/x64
Win8.1 RTM x86/x64

With Python 2.7.5, 3.2.3 and 3.3.2, x86 and x64 for each. (It helps to have a 
bit of practice with large test matrices, convenient access to clean VMs for 
lots of operating systems, and a couple of internal automation tools :) )

One issue I noticed is that if you've previously installed pip from source and 
it's in easy-install.pth, the version from the MSI won't be used when it is 
installed. The same applies to setuptools. (I used setuptools 0.9.8 and pip 
1.3.1 for testing this.)

In general there don't seem to be any other problems with it. I've noted some 
possible issues below, but since I don't know how much control you have over 
these, please don't take it personally if I'm pointing out things that cannot 
be changed.

- the default value for specifying a manual path (D:\PythonX) should probably 
be C:\PythonXY where XY is the version the installer is targeting.

- 64-bit versions of Python installed for all users are not detected. Are you 
planning a 64-bit version of this installer? (Specifying the path manually 
worked fine.)

- the RemoveFile table is incorrect for Python 3.x - there are no references to 
__pycache__ folders, only to .pyc and .pyo files in the same folder as their 
.py counterpart. As a result, uninstall is not clean for Python 3.2 and 3.3. In 
particular, Python 3.3 will show this (instead of No module named pip):

C:\ C:\Python33\python.exe -m pip
Traceback (most recent call last):
  File C:\Python33\lib\runpy.py, line 140, in _run_module_as_main
mod_name, loader, code, fname = _get_module_details(mod_name)
  File C:\Python33\lib\runpy.py, line 105, in _get_module_details
if loader.is_package(mod_name):
AttributeError: 'NamespaceLoader' object has no attribute 'is_package'

- choosing between 'All Users' and 'Just For Me' in the pip installer didn't 
seem to affect the install location.

- uninstalling Python before pip worked fine. (No problem here, just letting 
you know that I tried it :) )
- selecting both install options (Python from registry/custom path) and 
specifying the same path worked fine

Caveats:
- all machines were clean OS installs + Python from python.org. I didn't try 
installing Python from other sources
- I only tested upgrading pip with the installer on Windows 8, but I'm 
confident it will behave the same on all other platforms

All up, it looks great, and it's going to make things much easier for Windows 
users. Thanks for doing this.

Cheers,
Steve

-Original Message-
From: Distutils-SIG 
[mailto:distutils-sig-bounces+steve.dower=microsoft@python.org] On Behalf 
Of Donald Stufft
Sent: Thursday, 3 October 2013 0712
To: DistUtils mailing list
Subject: [Distutils] Test Windows Installers

Anyone who has a windows machine mind testing some installers for me?

These should install pip and setuptools:

https://dl.dropboxusercontent.com/u/45265381/MSI/pip/1.4/pip-1.4-py27.msi

https://dl.dropboxusercontent.com/u/45265381/MSI/pip/1.4/pip-1.4-py32.msi

https://dl.dropboxusercontent.com/u/45265381/MSI/pip/1.4/pip-1.4-py33.msi

Let me know?

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Test Windows Installers

[Steve Dower]

Donald Stufft wrote:
 On Oct 3, 2013, at 5:59 PM, Steve Dower steve.do...@microsoft.com wrote:
 
 I've done basic testing (install, pip install, pip list, pip uninstall,
 repair, uninstall) against:

 WinXP SP3 x86
 Vista SP2 x86/x64
 Win7 SP1 x86/x64
 Win8 RTM x86/x64
 Win8.1 RTM x86/x64

 With Python 2.7.5, 3.2.3 and 3.3.2, x86 and x64 for each. (It helps to
 have a bit of practice with large test matrices, convenient access to
 clean VMs for lots of operating systems, and a couple of internal
 automation tools :) )

 One issue I noticed is that if you've previously installed pip from
 source and it's in easy-install.pth, the version from the MSI won't be
 used when it is installed. The same applies to setuptools. (I used
 setuptools 0.9.8 and pip 1.3.1 for testing this.)
 
 What do you mean for this? That if you already have pip 1.3.1 installed and 
 you
 install from the MSI when it's over the installed version is still 1.3.1?

Both versions are installed, but `python -m pip` is going to select 1.3.1. (I 
didn't actually try Scripts\pip.exe, but since that's using pkg_resources I 
assume it will get the right one.)

I did the initial installs with the package setup.py files which produced a 
.egg file for setuptools and a .egg folder for pip. I'm not actually sure 
whether there's anything you could do about this from an installer...


Cheers,
Steve


 In general there don't seem to be any other problems with it. I've noted some
 possible issues below, but since I don't know how much control you have over
 these, please don't take it personally if I'm pointing out things that cannot 
 be
 changed.

 - the default value for specifying a manual path (D:\PythonX) should
 probably be C:\PythonXY where XY is the version the installer is targeting.
 
 Hmm, I'll see what I can do about this, I'm reusing routines from distutils to
 handle this but I may have some level of control over that.


 - 64-bit versions of Python installed for all users are not detected.
 Are you planning a 64-bit version of this installer? (Specifying the
 path manually worked fine.)
 
 Yea that was pointed out to me, I'll probably make 64bit installers since I
 don't think a 32bit installer can detect the 64bit versions.
 

 - the RemoveFile table is incorrect for Python 3.x - there are no references
 to __pycache__ folders, only to .pyc and .pyo files in the same folder as 
 their
 .py counterpart. As a result, uninstall is not clean for Python 3.2 and 3.3. 
 In
 particular, Python 3.3 will show this (instead of No module named pip):
 
 This is going to be an issue with distutils again, I'll see if I can control 
 it
 but it'd probably be a good idea to get this fixed in distutils too.
 

 C:\ C:\Python33\python.exe -m pip
 Traceback (most recent call last):
 File C:\Python33\lib\runpy.py, line 140, in _run_module_as_main
 mod_name, loader, code, fname = _get_module_details(mod_name) File
 C:\Python33\lib\runpy.py, line 105, in _get_module_details
 if loader.is_package(mod_name):
 AttributeError: 'NamespaceLoader' object has no attribute 'is_package'

 - choosing between 'All Users' and 'Just For Me' in the pip installer didn't
 seem to affect the install location.
 
 AFAIK distutils added that, no idea what it does or means. I'll see if I can
 remove it.
 

 - uninstalling Python before pip worked fine. (No problem here, just
 letting you know that I tried it :) )
 - selecting both install options (Python from registry/custom path)
 and specifying the same path worked fine

 Caveats:
 - all machines were clean OS installs + Python from python.org. I
 didn't try installing Python from other sources
 - I only tested upgrading pip with the installer on Windows 8, but I'm
 confident it will behave the same on all other platforms

 All up, it looks great, and it's going to make things much easier for Windows
 users. Thanks for doing this.

 Cheers,
 Steve

 -Original Message-
 From: Distutils-SIG
 [mailto:distutils-sig-bounces+steve.dower=microsoft@python.org] On
 Behalf Of Donald Stufft
 Sent: Thursday, 3 October 2013 0712
 To: DistUtils mailing list
 Subject: [Distutils] Test Windows Installers

 Anyone who has a windows machine mind testing some installers for me?

 These should install pip and setuptools:

 https://dl.dropboxusercontent.com/u/45265381/MSI/pip/1.4/pip-1.4-py27.
 msi

 https://dl.dropboxusercontent.com/u/45265381/MSI/pip/1.4/pip-1.4-py32.
 msi

 https://dl.dropboxusercontent.com/u/45265381/MSI/pip/1.4/pip-1.4-py33.
 msi

 Let me know?

 -
 Donald Stufft
 PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9
 3372 DCFA

 
 
 -
 Donald Stufft
 PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

[Steve Dower]

Oscar Benjamin wrote:
 Paul wrote:
 Given that the installer includes the py.exe launcher, if you leave the
 defaults, then at a command prompt python doesn't work. But that's fine,
 because py does. And if you have multiple versions of Python installed, you
 don't *want* python on PATH, because then you have to manage your PATH. Why
 bother when py -2.7 or py -3.3 does what you want with no path 
 management?
 Once you want any *other* executables, though, you have to deal with PATH
 (especially in the multiple Pythons case). That is a new issue, and one that
 hasn't been thought through yet, and we don't have a good solution.

 From a user perspective I think that 'py -3.4 -m pip ...' is an improvement as
 it means I can easily install or upgrade for a particular python installation 
 (I
 tend to have a few). There's no need to put Scripts on PATH just to run pip. I
 think this should be the recommended invocation for Windows users.

Crazy idea:

py install other args
(or 'py --install ...', but I think 'py install' is currently invalid and could 
be used?)

which the launcher executes identically to:

py -m pip install other args

(Implicitly extended to other relevant commands... I'm not proposing a complete 
list.)

Pros:
* allows implicit bootstrapping on first use (from a bundled pip, IMO, in case 
no network is available)
* multiple Python versions are handled nicely and consistently ('py -3.3 
install ...')
* can minimize officially supported API surface (as Paul described at the start 
of this thread)
* pip becomes an internal implementation detail that can be entirely replaced
* one less character of typing (slightly tongue-in-cheek, but some people count 
:) )

Cons:
* doesn't apply on *nix (or does/could it?)
* requires the most new code of any of the options
* more difficult to update the launcher than a user-installed package
* others that I can't think of because I'm suffering from confirmation bias?

Thoughts?

Cheers,
Steve
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig