[Distutils] Re: Is ensurepip still a thing?

[Chris Barker - NOAA Federal via Distutils-SIG]

> IIRC, ensurepip by design doesn't go to the internet , so it will only
> ever upgrade to the version bundled with Python

Now I’m really confused — if pip is already bundled with Python, then
what is ensurepip for ?!?!

Or really, the question at hand: should a user starting from scratch
with a python.org install of 3.7 run ensurepip?

Or can they just go straight to:

Python3 -m pip install —upgrade pip

?!?

-CHB


> from the docs "To
> ensure the installed version of pip is *at least as recent as the one
> bundled with ensurepip*, pass the --upgrade option" [emphasis mine]).
> To get the latest available version, you should do `python -m pip
> install --upgrade pip` (better than `pip install...` as it works on
> Windows) as you mentioned.
>
> Paul
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/XGQ7HMBNLSRG37IMIQ5WXAMEDS6UNLJW/

[Distutils] Re: Packaging Advice for EFF's Certbot

[Chris Barker - NOAA Federal via Distutils-SIG]

I agree that you are probably best off integrating with the system
packaging system in this case.

But if you do want to deploy and app with all its dependencies in a
controlled environment, conda constructor May make that easy:

https://github.com/conda/constructor

-CHB



Sent from my iPhone

On Jul 26, 2018, at 4:20 AM, Nick Coghlan  wrote:

On 25 July 2018 at 12:39, Nathaniel Smith  wrote:

On Jul 24, 2018, at 4:36 AM, Nick Coghlan  wrote:


However, there *are* folks that have been working on allowing

applications to be defined primarily as Python projects, and then have

the creation of wrapper native installers be a pushbutton exercise,

rather than requiring careful human handholding.


But it sounds like they also want to be able to install/remove/upgrade

*parts* of the Python project, for their plugin support. And maybe

upgrade the Python interpreter as well. Do any of these tools allow

that? That's the thing that really made me think about conda.


Right, that's why my suggestion was for a two layer solution (native
packaging of a base platform integration layer via fpm, combined with
pip for plugin management within that base environment), akin to the
way Linux distro packages of Firefox and Chromium still leave the
browser to do its own plugin management.

That way the fpm-built native package can depend on any required
system packages, as well as lay out the base virtual environment in
/opt. In many ways, it's the same thing that certbot-auto is already
doing, it's just replacing the current directly downloaded shell
script with native Linux packages built with fpm.

You can certainly do the same thing with conda instead (as per [1]),
but given that the target audience for certbot includes professional
Linux sysadmins, being able to integrate with the native system
package manager seems to be an actively desired feature rather than an
unwanted hassle. So while I'd agree conda is well worth a look as a
potential helper for environment management within the /opt directory,
in this particular case I don't think it's going to be a suitable
replacement for offering native packages as the core update mechanism
for the base platform.

Cheers,
Nick.

[1]
http://www.curiousefficiency.org/posts/2016/09/python-packaging-ecosystem.html#platform-management-or-plugin-management

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/KJQNVRZJ4VTIO2IPAUVO4MSWDEK6WULI/
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-le...@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at 
https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/VJBMVFKAH6FYYBMHXEJSBMH5G3UQHJJJ/

[Distutils] Or in version spec...

[Chris Barker - NOAA Federal]

Sorry to lose the thread — lousy iPhone mail app...

Conda supports or in its meta.yaml format:

https://conda.io/docs/user-guide/tasks/build-packages/package-spec.html#build-version-spec

Maybe look to that for prior art?

And it would be mildly less confusing to have consistency between the
systems.

-Chris


Sent from my iPhone
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outstanding questions for PEP 541: Package Index Name Retention

[Chris Barker - NOAA Federal]

> I don’t know how much we can or should define “being used”. For instance,
every package in existence is going to be downloaded via mirroring, so we
can’t go by pure download counts (and some mirrors just use pip to do their
downloading) and we’re going to need to interpret the download counts to
some degree to try and gauge “usage”.

One idea— create a couple of “dummy” packages that do absolutely nothing.
Then the download counts on those would give you a baseline for “downloaded
but not used”

-CHB

Ultimately I think the underlying question we’re answering is whether names
on PyPI are a community good or a personal good, and this PEP more or less
states they are a community good


Which they absolutely are. PyPi is a community project — the names on pypi
are just that — names on pypi. If you don’t want to be part of that
community, you can use any name you want.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [proposal] shared distribution installations

[Chris Barker - NOAA Federal]

, the behaviour i aim for would be moslty like
virtualenv but without the file duplication.


For what it’s worth, conda environments use hard links where possible, so
limiting duplication...

Maybe conda would solve your problem...

-CHB







I beleive nix could also benefit from parts of such a mechanism.

-- Ronny


Am Montag, den 30.10.2017, 20:35 +0100 schrieb Freddy Rietdijk:

Hi Ronny,


What you describe here is, as you know, basically what the Nix

package manager does. You could create something similar specifically

for Python, like e.g. `ied` is for Node [2], or Spack, which is

written in Python. But then how are you going to deal with other

system libraries, and impurities? And you will have to deal with

them, because depending on how you configure Python packages that

depend on them (say a `numpy`), their output will be different. Or

would you choose to ignore this?


Freddy


[1] https://nixos.org/nix/

[2] https://github.com/alexanderGugel/ied

[3] https://spack.io/


On Mon, Oct 30, 2017 at 8:16 PM, RonnyPfannschmidt  wrote:

Hi everyone,


since a while now various details of installing python packages in

virtualenvs caused me grief


a) typically each tox folder in a project is massive, and has a lot

of

duplicate files, recreating them, managing and iterating them takes

quite a while

b) for nicely separated deployments, each virtualenv for an

application

takes a few hundred megabytes - that quickly can saturate disk

space

even if a reasonable amount was reserved

c) installation and recreation of virtualenvs with the same set of

packages takes quite a while (even with pip caches this is slow,

and

there is no good reason to avoid making it completely

instantaneous)


in order to elevate those issues i would like to propose a new

installation layout,

where instead of storing each distribution in every python all

distributions would share a storage, and each individual

environment

would only have references to the packages that where

"installed/activated" for them


this would massively reduce time required to create the contents of

the

environments and also the space required


since blindly expanding sys.path would lead to similar performance

issues as where seen with setuptools/buildout multi-version

installs,

this mechanism would also need a element on sys.meta_path that

handles

inexpensive dispatch to the toplevels and metadata files of each

packages (off hand i would assume linear walking of hundreds of

entries

simply isn't that effective)


however there would be need for some experimentation to see what

tradeoff is sensible there


I hope this mail will spark enough discussion to enable the

creation of

a PEP and a prototype.



Best, Ronny




___

Distutils-SIG maillist  -  Distutils-SIG@python.org

https://mail.python.org/mailman/listinfo/distutils-sig



___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Extracting distutils into setuptools

[Chris Barker - NOAA Federal]

 distutils works fine for its original purpose (building components for
 the system Python in Linux distros),

What does Linux have to with it? In the eagle days, I found it most
helpful for Windows, actually. And it's very helpful for OS-X as well.

It was also great for pure python bundling and installing.

But regardless of original intent, it's clear that distutils has not
turned out to be as cleanly extendible as is needed.

Setuptools has its issues as well -- personally it has always driven
me crazy with its mingling of run-time vs build-time vs install time
functionality.

I think it would be helpful to be explicit about the final goals
before talking about whether and how to fork distutils.

What I'd love to see is a build tool that is compatible with the
"modern" packaging paradigm-- pip, pypi, wheels. ( and other systems
like rpm, conda, )

Essentially setuptools without the cruft. ( no easy install, pkg_resources, ...)

This could be maintained as a third party package, like the current
setuptools. If/when it becomes ubiquitous, an "ensure" style module
could be added to the stdlib.

So how do we get there? I would think a fork and merge of both
setuptools and distutils would make sense. Preserve the building part
of the API, so folks could simply replace the import line an their
setup.py files to use the new tool.

Distutils and setuptools remain in maintainable only mode for probably
a long time :-(

-CHB



 so we still need to avoid
 breaking that. setuptools is only essential if you want full support
 for modern *Python* level packaging features (PEP 376 install
 metadata, venv compatibility, wheel files, etc), and a lot of Linux
 system components simply don't worry about those things, and rely on
 their system level equivalents instead (e.g. the RPM/deb databases,
 chroots and containers, RPM/deb files)

 However, what *could* be interesting is a proposal to move distutils
 to the "ensurepip" model, where rather than maintaining distutils
 directly as part of CPython, the CPython build process instead runs
 setuptools/distutils from a bundled wheel file. Doing that would
 entail having setuptools actually start installing a copy of distutils
 into site-packages: older CPython releases would ignore it by default
 (since the stdlib version would shadow it), while 3.7+ would offer
 either "python3 -m ensuredistutils" or "python3 -m ensuresetuptools"
 (bikeshed to be painted via the PEP process) to install the
 setuptools-provided version.

 I'm not claiming actually doing that would be particularly easy - I
 just think it's the most viable path to get us away from the current
 version coupling between the build infrastructure in distutils and the
 runtime support infrastructure in the rest of the standard library,
 and to avoid maintaining two distinct copies of distutils indefinitely
 (one in the stdlib, one in setuptools).

 That approach wouldn't even entail any *new* bundling at the CPython
 level, as while it's currently formally an implementation detail
 (pending potential removal in a post-PEP-517 world), setuptools is
 already bundled as part of the support infrastructure for ensurepip:
 https://github.com/python/cpython/tree/master/Lib/ensurepip/_bundled

 Cheers,
 Nick.

 --
 Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
>>>
>>>
>>> ___
>>> Distutils-SIG maillist  -  Distutils-SIG@python.org
>>> https://mail.python.org/mailman/listinfo/distutils-sig
>>>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Accepting PEP 541?

[Chris Barker - NOAA Federal]

This looks great: thanks for moving it along!


Minor notes (all copy editing):

It could use some editing to bring it into the present:

"""
Existing package repositories such as CPAN [3]
, NPM [4]
, and GitHub [5]
 will be investigated as
prior art in this field.
"""

Probably need a time scale on:


   - no activity from the owner on the project's home page.

"""

Minor edit:
"the candidate is able to demonstrate own failed attempts to contact the
existing owner"

To

"the candidate is able to demonstrate their own failed attempts to contact
the existing owner"

(Adding "their" -- or his/her)

Same under "abandoned".


-CHB

Sent from my iPhone

On Sep 8, 2017, at 12:14 PM, Lukasz Langa  wrote:

I gathered and responded to feedback on this early this year. I think the
PEP is ready to be accepted.

https://www.python.org/dev/peps/pep-0541/

- Ł

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker - NOAA Federal]

One thing to keep in mind is that there are quite a few projects on pypa
with pure python source distributions uploaded that will not be updated and
people may still desire to use. We want pip to be able to still build and
install them.


That is the challenge!

But the motivating use case here was a package with a c extension that was
optionally built.

That's the rare case.

Regular old pure-python packages are the usual case, and a lot easier.

-CHB

2017-08-31 16:29 GMT-05:00 Chris Barker :

> On Thu, Aug 31, 2017 at 11:03 AM, Nathaniel Smith  wrote:
>
>> > Surely the build system should know how to correctly name the wheel it
>> builds.
>>
>> It's probably worth mentioning the specific problem that motivated pip
>> to start doing this.
>>
>> It used to be standard, and is still quite common, for setup.py
>> scripts to contain stuff like:
>>
>> install_requires = [...]
>> if sys.version_info < (3, 4):
>> install_requires += [...]
>> if platform.python_implementation() == "PyPy":
>> install_requires += [...]
>>
>> setup(..., install_requires=install_requires)
>>
>> This kind of logic in setup.py worked fine in the old days when all
>> you did was 'setup.py install', but then wheels came along
>
>
> And indeed, setuptools originally used easy_install, which was part of
> setuptools...
>
> and
>> retroactively turned lots of setup.py scripts from working into
>> broken. The problem is that with this kind of setup.py,
>>
>
>
>> But it will take a while for existing setup.py files transition to
>> using those, and in the mean time pip can't assume that a random wheel
>> generated by 'setup.py bdist_wheel' has accurate Python tags.
>>
>
> This was my original point -- I understand that we want "pip install" to
> continue to work for, hopefully, everything it works for now.
>
> But I do think we should be clear about what is a hack for backward
> compatibility, and what is part of the designed functionality.
>
> Sorry to be poking at all this from the fringes (Not having been all that
> involved in a very long discussion), it's just that the whole
>
> distutils--setuptools--pip--distribute--setuptools--pip
>
> stack has a LOT of legacy cruft, and I'm  concerned that the efforts for
> backward compatibility may end up leading us to another poorly de-coupled
> design.
>
>
>> Hopefully new legacy-free backends will get this right from the start.
>>
>
> exactly -- let's keep the "backward compatibility hack" labels clear!
>
> -CHB
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] PEP 517 again

[Chris Barker - NOAA Federal]

> that neither pip nor the setuptools backend should not change the tags
> it applies to wheels by default).

I'm a bit confused -- are we talking about the backwards compatible
path to the future -- or the end-game?

In short -- I'm sure we'll have to do some hacky stuff to keep
backwards compatibility, but the end game should be a clean separation
of concerns :

Pip (or any front end) should never "build a wheel", and it certainly
shouldn't have to know enough about what's in a wheel to be re-naming
it for generic python vs cpython.

The package manager should manage the package, not built it, or change it.

Surely the build system should know how to correctly name the wheel it builds.

As to using pip to build wheels -- there is good reason to do that
now, but in s post PEP 517 world, one would call the build system
directly to build a wheel-- after all, all pip should be  doing is
calling the build system anyway.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Distuils MSVC link failure between libraries

[Chris Barker - NOAA Federal]

I'm trying to create multiple C++ extensions that have dependencies between
them.

This works fine on Linux with gcc, but I get link failures on Windows with
MSVC due to this



Is that the only issue? MSVC and FCC linking is pretty different-- I know I
tried to do something like this (a few years ago) and it didn't work -- I
think because the symbols I needed weren't exported, or something like that.

If precompiled extensions are to be published with MSVC and expected to be
linked against (maybe this is not recommended?)


Indeed, it is not recommended-- onhe passive sense -- I.e. the system isn't
designed to support that.

IIUC, while extension modules are dlls, they are specifically designed to
be called from cpython, not as general purpose libraries to be called from
other C code.

The solution is to build a dill with all the C/C++ code you need, and then
have all your modules link against that dll.

Proof of concept



If that works, and doesn't cause other problems, (I am not qualified to
assess that), it could be handy. There are times I would have liked
distutils to be able to build a dll for me that could be used by other
modules.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Which commercial vendor?

[Chris Barker - NOAA Federal]

> On Mar 30, 2017, at 1:53 AM, Thomas Güttler  
> wrote:
>
>
> My frustration has reached a limit. Yes, I am willing to pay money.
>
> Which vendor do you suggest to give me a reliable package management?

You may want conda -- it's an open source project, and you can get
commercial support through Continuum Analytics.

Though conda, and Continuum was born in the data science / scientific
computing world, there is nothing about conda specific to data
science. It's just that computational computing poses extra challenges
for packaging -- but the easy stuff is still easy.

And the conda-forge project provides an open source way to support a
larger community.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] install questions and help requested. ---pyautogui

[Chris Barker - NOAA Federal]

This is really a list for discussing development of distribution
tools, rather than help on basic usage. But;

>
>  >>> pip install pyautogui
> SyntaxError: invalid syntax

This looks like you are trying to run pip at the Python prompt. Pip is
designed to be run st a system command line ("DOS prompt")

Try the same command there.

-CHB


>
> Sincerely,
> Michael G. Strain  Jr.
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Maintaining a curated set of Python packages

[Chris Barker - NOAA Federal]

>
>
> I think it's unfair to describe these efforts as a "kludge";

I was specifically referring to using wheels to deliver C libs--
pip+wheel were not designed for that. But I don't mean to offend,
there has been a lot of great work done, and yes, the situation has
much improved as a result.

There are times when a kludge is called for!

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Deprecating little used file types/extensions on PyPI?

[Chris Barker - NOAA Federal]

>
> If we're going through all this trouble, isn't it better just to jump to .zip 
> files like every other distribution format in existence?

Yes. :-)

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What role to eggs still play?

[Chris Barker - NOAA Federal]

Thanks, I think I'm getting it.

About the toml file... the *-info metadata is a compiled artifact,
according to all the existing Python packages. Most sdists even have a
*.egg-info directory.


If it's a compiled artifact, then shouldn't it NOT be in a source dist?

It is inconvenient if you want to know the true dependencies without
running setup.py.


Isn't that what the toml file is for?

I think we are stuck with it, and it's not all bad - if there is some
useful metadata that doesn't affect the dependency resolver, and it is "too
static" or too cumbersome to write out by hand, there's a place to do that.


I'm trying to imagine what that info would be,  but I suppose there could
be meta data about a package that is generated at build time -- maybe info
about how it was built, for instance.

Hmm -- maybe you could put info in there about non-Python shared libs it's
linked to, for instance.

Eggs are the only way to add a zipped distribution to PYTHONPATH and have
> setuptools find the metadata
>

Can pip find it in a zipped package? Remember,  I don't care to support
setuptools only features anyway :-)

. Eggs are used by buildout, especially in the unzipped into a directory
> form
>
And they could still be used for their originally designed use as a plugin
> format.
>

If pkg_resources gets spun off, would it support that?

 *.egg-info or EGG-INFO is the predecessor of the *.dist-info format
> designed in PEP-376. You get *.egg-info whenever you install something
> with setuptools without going through bdist_wheel.
>

So setuptools_lite would write a  dist_info.

I take it pip looks for both?
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Outdated packages on pypi

[Chris Barker - NOAA Federal]

Getting to this thread late, but it didn't seem that was resolved in
the least, so I'll as my $0.02

> That overall got me thinking about namespace pollution in pip, that
> once something is pushed in, it's like to stay there forever.

This REALLY is a problem, and one that will only get worse. It would
be nice to work out an approach before it gets too bad.

> I'd rather see something similar to Linux distributions where there's
> a curated repository "core"

As pointed out, that's not going to happen.

> and a few semi-official, like "extra" and
> "community," and for some, "testing."

But namespaces like these could still be implemented without curation.

Right now, the barrier to entry to putting a package up on PyPI is
very, very low. There are a lot of 'toy', or 'experimental', or 'well
intentioned but abandoned' projects on there.

If there was a clear and obvious place for folks to put these packages
while they work out the kinks, and determine whether the package is
really going to fly, I think people would use it.

And we could have mostly-automatic policies about seemingly abandoned
projects as well -- moving them to the "unmaintained" channel, or

As for the 'deadman's switch' idea -- that's a great idea. Sure, there
are projects with no active maintainer that people rely on, but almost
be definition, if folks care about it, we'd be able to find someone
willing to put their finger on the dads man's button once a year

As for issues like setuptools and PIL, where there is a shift in
maintenance if a large, high profile package, we really need a
benevolent oligarchy for PyPA that can resolve these issues by hand.
As pointed out -- this really doesn't come up often.

However, in these discussions, I've observed a common theme: folks in
the community bring up issues about unmaintained packages, namespace
pollution, etc. the core PyPA folks respond with generally well
reasoned arguments why proposed solutions won't fly.

But it's totally unclear to me whether the core devs don't think these
are problems worth addressing, or think they can only be addresses
with major effort that no one has time for.

If the core devs think it's fine and dandy like it is, we can all stop
talking about it.

By the way, there is an experiment underway with a "curated" community
package repository for conda packages:

https://conda-forge.github.io

It's semi-curated, in the sense that only the core devs can approve a
package being added, but once added, anyone can maintain it.

It's going very well, but I'm not sure how it's going to scale. So
far, 900 or so packages, 80 or so maintainers. Which I think is very
impressive for a young system, but still a lot smaller than it could
be.

But I think PyPA should keep an eye on it -- I'm sure there will be
lessons learned.

-CHB






> A name foobar resolves to core/foobar- if that exists, and if
> not some subset of other repositories is used.
> This way, an outdated package can be moved to another repo without
> breaking install base.
>
> In fact, curation without namespaces will already be pretty good.
>
> d.
>
>> On 13 July 2016 at 19:24, Jim Fulton  wrote:
>>> On Tue, Jul 12, 2016 at 7:55 AM, Dima Tisnek  wrote:
>>> Hi all,
>>>
>>> Is anyone working on pruning old packages from pypi?
>>>
>>> I found something last updated in 2014, which, looking at the source
>>> appears half-done.
>>> Github link doesn't work any longer, no description, etc.
>>>
>>> I managed to find author's email address out of band, and he responded
>>> that he can't remember the password, yada yada.
>>>
>>> I wonder if some basic automation is possible here -- check if url's
>>> are reachable and if existing package satisfies basic requirements,
>>> failing that mark it as "possibly out of date"
>>
>> I'm curious why you view this as a problem that needs to be solved?
>>
>> - Do you want to take over the name yourself?
>>
>> - Are you afraid someone will stumble on this package and use it?
>>
>> Something else?
>>
>> Jim
>>
>> --
>> Jim Fulton
>> http://jimfulton.info
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] comparison of configuration languages

[Chris Barker - NOAA Federal]

​Not asking for any change but has anyone looked at libconfig
? ​It looks
quite interesting: simple grammar and nesting support. What do you think of
it


As pointed out, it's a C lib. But as we all like writing tools, it wouldn't
be very hard to write a Python parser for the format.

But it's a bit C-y for my taste, and yet another configure language? Really?

-CHB



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Name arbitration on PyPI

[Chris Barker - NOAA Federal]

Another high profile example of such a project: PIL.


Was PIL ever on PyPi? Anyway, yup, the solution there was to fork give it s
new name -- Pillow was born.

CHB


19.04.2016, 00:56, Chris Barker kirjoitti:

On Mon, Apr 18, 2016 at 2:31 PM, Ian Cordasco 
wrote:

> >> 1.� PyYAML is a package that would be de-registered in such a
> scheme.� It
>
�

> > and you don't think ANYONE would be willing to take on the miniscule
> amount
> > of work to maintain the name? Plus there would be any number of other
> > schemes for determining whether a project name is abandoned.
>
> I have in fact offered but the author refuses to accept help from
> anyone. They're also the author of the C library (libyaml) and they do
> not maintain that either. It's actually quite frustrating as someone
> who wants to fix some of the numerous bugs in the library + improve it
> and add support for YAML 1.2 which is years old at this point.


Interesting third case I hadn't considered -- the original author is still
"active", but not actually maintaining the software or accepting help. I
don't think there is anything PyPi policy can do about that -- too bad.

Time for a fork?�

-CHB


�
-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR � � � � � �(206) 526-6959�� voice
7600 Sand Point Way NE ��(206) 526-6329�� fax
Seattle, WA �98115 � � ��(206) 526-6317�� main reception

chris.bar...@noaa.gov


___
Distutils-SIG maillist  -
Distutils-SIG@python.orghttps://mail.python.org/mailman/listinfo/distutils-sig


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The mypy package

[Chris Barker - NOAA Federal]

> Though I do wonder how effective that would be in this case.  For all we 
> know, in the case of mypy, the maintainer is simply ignoring someone else who 
> is trying to take the name they registered.  (I get emails all the time for 
> people trying to get me to sign over my domain names;

Domain names are a different system -- you need to maintain your registration.

PyPi names, on the other hand, are all too easy to setup, and then
completely ignore, maybe even forget you used it.

We really should have SOME way to determine if a PyPi name has been
abandoned. Or even be proactive--PyPi names must be maintained in SOME
way, perhaps:

Push a change or update at least once a year (or some other interval).

Or

Respond to some sort of "do you still want this" email. At least once a year.

If neither of these occurs, then we could have a deprecation period.

Details aside, as PyPi continues to grow, we really need a way to
clear out the abandoned stuff -- the barrier to entry for creating a
new name on PyPi is just too low.

This is all too late for MyPy, but it has certainly come up before,
and will again, more and more.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What's up with PyPi maintenance?

[Chris Barker - NOAA Federal]

b) Is there a serious lack of folks available to address such issues?
>

This is the correct answer. I can't even keep on top of the request from
people wanting to take over packages.


if (b), should we make a concerted effort to recruit new folks to assist
> PyPi maintenance / development???
>

That would be nice, but would be pointless until Warehouse goes live.


When do we expect that? A lot of people rely on the current system, we
really need to find a way to maintain it 'till it's replaced.

CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to get pip to really, really, I mean it -- rebuild this damn package!

[Chris Barker - NOAA Federal]

>>  Requirement already satisfied (use --upgrade to upgrade): gsw==3.0.3 from
>> file:///Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3 in
>> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3
>
> I think this is saying that pip thinks it has found an
> already-installed version of gsw 3.0.3 in sys.path, and that the
> directory in your sys.path where it's already installed is
>
> /Users/chris.barker/miniconda2/conda-bld/work/gsw-3.0.3

That is the temp dir conda sets up to unpack downloaded files, and do
its work in -- hence the name. I'll look and see what's there. I'm
pretty sure conda build starts out with an empty dir, however. And
that dir should not be on sys.path.

> I think this means that that directory is (a) in sys.path, and (b)
> contains a .egg-info/.dist-info directory for gsw 3.0.3. Part (a)
> seems weird and broken.

Indeed. And I get the same symptoms with a clean environment that I've
set up outside conda build. Though with the same source dir. But with
conda build, it's a fresh unpack of the tarball.

> Do you have "." in your PYTHONPATH or anything like that?

God no!

> Don't know why it seems to be building a wheel for it, if it already
> thinks that it's installed... this is also odd.

Yes it is. But it doesn't install it :-(

>
> $PYTHON -m pip install --no-cache-dir --upgrade --force-reinstall ./
>
> ? Though I'd think that -I would have the same affect as --force-reinstall...
>
So did I, and I think I tried --force-reinstall already, but I will again.

> (It doesn't look like the cache dir is your problem here, but you do
> probably want to use --no-cache-dir anyway just as good practice, just
> because you don't want to accidentally package up a stale version of
> the software that got pulled out of your cache instead of the version
> you thought you were packaging in the tree in front of you.

Exactly. Doesn't seem to make a difference, though.

> Also, I think it's a bug in pip that it caches builds of source trees
> -- PyPI can enforce the rule that each (package name, version number)
> sdist is unique, but for a work-in-progress VCS checkout it's just not
> true that (package name, version number) uniquely identifies a
> snapshot of the whole tree. So in something like 'pip install .', then
> requirement resolution code should treat this as a special requirement
> that it wants *this exact tree*, not just any package that has the
> same (package name, version number) as this tree; and the resulting
> wheel should not be cached.

Absolutely! In fact, I'll bet that approach is the source of the
problem here. If not automagically, there should be a flag, at least.

However, what seems to be happening is that pip is looking outside the
current Python environment somewhere to see if this package needs to
be installed. It may be something that works with virtualenv, but
doesn't with conda environments for some reason.

I guess on some level pip simply isn't designed to build and install
from local source :-(

In the end, I'm still confused: does pip install give me anything that:

setup.py install single-version-externally-managed

Doesn't? Other that support for non-setuptools installs, anyway.

CHB


> I don't know if there are any bugs filed
> in pip on this...)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Chris Barker - NOAA Federal]

(e.g. by bumping up
>> the base ABI from CentOS 5 to CentOS 6).
>
> The problem with this is that python 2.7 is going to be supported and
> widely used until well past the EOL of CentOS 5, and maybe even past
> the EOL of CentOS 6

Given that we're starting now ( not a year or two ago) and it'll take
a while for it to really catch on, we should go CentOS 6 ( or
equivalent ) now?

CentOS5 was released in 2007! That is a pretty long time in computing.

Just a thought, we'll be locked into it for a while, yes?

-CHB


> (in 2020). On Linux, unlike Windows, the whole
> system ABI underneath Python evolves in a backwards-compatible way, so
> there's no anchor like the MSVC CRT that's tying us to old ABIs.
>
> (And on OS X, the OS X version number is encoded as part of the
> platform tag, similar to the proposal for manylinux1/2/...)
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Chris Barker - NOAA Federal]

> So without a plan to provide all that stuff -- I"m not sure of the utility
> of this -- how are you gong to get PIL/Pillow to work? statically link up
> the ying-yang? Not sure the linux world will take to that.
>

We can explain how things work in details for some packages, but the main
rationale for the PEP list is that this is a list that works in practice.
It has worked well for us at Enthought for many years, and it has for
(Ana)conda as well.


My point was that it works in those cases because Enthought and Continuum
provide a bunch of (often hard to build) packages that provide all the
extra stuff.

Maybe the community will spring forth and do that -- I'm skeptical because
I tried to to that for years for OS-X and it was just too much to do. And
the infrastructure was there.

Before pip and wheel there were mpkgs on OS-X, and repo's of toms for Linux
years before that -- but always the result of a couple people's heroic
efforts.

Maybe the infrastructure has improved, and the community grown enough, that
this will all work. We'll see.

CHB



Between both distributions, we are talking about millions of installs over
the year, on many different systems.

David
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] draft PEP: manylinux1

[Chris Barker - NOAA Federal]

I went through this list and compiled manylinux1 wheels for each of the top
15 projects in the
list (py35).

The wheels are here, if you're interested
http://stanford.edu/~rmcgibbo/wheelhouse


Cool! Are the non-manylinux dependencies all statically linked?

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Installing packages using pip

[Chris Barker - NOAA Federal]

How it works in the terminal is less important to me than how it works in
IDLE though; being able to teach how to use Python through IDLE (deferring
discussion of terminals etc) is useful for introductory programming classes.
Personally, I don't use IDLE for teaching,  but do use iPython. But if we
have a way to call pip from a Python REPL, it really should work in the
standard REPL. Though still a good idea to have IDLE specific and iPython
specific ways to install packages within those environments. I like the

%pip

Idea for iPython -- and I'm pretty sure the kernel can be restarted.
Certainly in a notebook.

As for the plain REPL, maybe a warning that you need to restart after an
upgrade would be enough.

Though I suspect that Window's aggressive file locking will put the kibosh
on in-place upgrades :)

CHB




--
Oscar

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Chris Barker - NOAA Federal]

 > If we waved our hands and were able to magically make Python package
> management perfect, what would that look like?

well, I think the command would be:

python install package_name

I know there are good reasons to keep package installer development out of
core, but if have ensurepip-- we could do this.

CHB

Would that kind of discussion even be valuable?

On Tue, Nov 10, 2015, 6:22 PM Nathaniel Smith  wrote:


I totally get why people dislike the ergonomics of 'python -m pip',
but we can also acknowledge that it does solve a real technical
problem: it strictly reduces the number of things that can go wrong,
in a tool that's down at the base of the stack.


___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] The future of invoking pip

[Chris Barker - NOAA Federal]

> In this situation, if pip's default for finding the python environment
> were to search the path for 'python' instead of using sys.executable,

One trick here -- PATH may not be the same everywhere. For instance,
on OS-X, the environment GUI programs get is entirely independent of
the shell. So, as a rule, PATH will not be "right". I don't know if
anyone tries to run pip from an IDE, but it might be worth keeping in
mind-- this issue comes up all the time with setting the Python used
by an IDE.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Chris Barker - NOAA Federal]

> While I understand what you're trying to achieve (and I'm in favour,
> in general) it should be remembered that pip's core goal is installing
> packages - not being a component of a development workflow.

Yes -- clear separation of concerns here!

So what IS supposed to be used in the development workflow? The new
mythical build system?

This brings.  me back to my setuptools-lite concept -- while we are
waiting for a new build system, you can use setuptools-lite, and get a
setup.py install or setup.py develop that does what it's supposed to
do and nothing else

OK, I'll go away now :-)

-Chris


> We absolutely need to make pip useful in the development workflow type
> of situation (that's why pip install -e exists, after all). But I
> don't think it's so much pip "trying to be too clever" as incremental
> rebuilds wasn't the original use case that "pip install ." was
> designed for. What we'll probably have to do is be *more* clever to
> special case out the situations where a development-style support for
> incremental rebuilds is more appropriate than the current behaviour.
>
> Paul
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Platform tags for OS X binary wheels

[Chris Barker - NOAA Federal]

On Nov 6, 2015, at 3:57 PM, Robert McGibbon  wrote:

I'm using the Python from the Miniconda installer with py35 released last
week.


Then you should not expect it to be able to find compatible binary wheels
on PyPi.

Pretty much the entire point of conda is to support Numpy and friends. It's
actually really good that it DIDN'T go and install a binary wheel.

You want:

conda install numpy

Trust me on that :-)

There are some cases where pip installing a source package into a conda
Python is fine -- but mostly only pure-Python packages.

-CHB



What does the python.org installer build for 10.6+ return for
`distutils.util.get_platform()`?

-Robert

On Fri, Nov 6, 2015 at 2:50 PM, Ned Deily  wrote:

> In article
> ,
>  Robert McGibbon  wrote:
> > I just tried to run `pip install numpy` on my OS X 10.10.3 box, and it
> > proceeds to download and compile the tarball from PyPI from source (very
> > slow). I see, however, that pre-compiled OS X wheel files are available
> on
> > PyPI for OS X 10.6 and later.
> >
> > Checking the code, it looks like pip is picking up the platform tag
> through
> > `distutils.util.get_platform()`, which returns 'macosx-10.5-x86_64' on
> this
> > machine. At root, I think this comes from the
> MACOSX_DEPLOYMENT_TARGET=10.5
> > entry in the Makefile at `python3.5/config-3.5m/Makefile`. I know that
> this
> > value is used by distutils compiling python extension modules --
> presumably
> > so that they can be distributed to any target machine with OS X >=10.5 --
> > so that's good. But is this the right thing for pip to be using when
> > checking whether a binary wheel is compatible? I see it mentioned
> >  in PEP 425, so perhaps
> > this was already hashed out on the list.
>
> Are you using an OS X Python installed from a python.org installer?  If
> so, be aware that there are two different OS X installers on Python.org
> 
> for each current release.  One is intended for 10.5 systems, although it
> will work on later OS X systems.  The other is for 10.6 and later
> systems.  Unless you have a need to run on 10.5 or build something that
> works on 10.5, download and use the 10.6+ installers instead.  Then the
> existing whls for products like Numpy should work just fine.
>
> --
>  Ned Deily,
>  n...@acm.org
>
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[Chris Barker - NOAA Federal]

>> I'm not talking about in place installs, I'm talking about e.g. building a
>> wheel and then tweaking one file and rebuilding -- traditionally build
>> systems go to some effort to keep track of intermediate artifacts and reuse
>> them across builds when possible, but if you always copy the source tree
>> into a temporary directory before building then there's not much the build
>> system can do.

This strikes me as an optimization -- is it an important one?

If I'm doing a lot of tweaking and re-running, I'm usually in develop mode.

I can see that when you build a wheel, you may build it, test it,
discover an wheel-specific error, and then need to repeat the cycle --
but is that a major use-case?

That being said, I have been pretty frustrated debugging conda-build
scripts -- there is a lot of overhead setting up the build environment
each time you do a build...

But with wheel building there is much less overhead, and far fewer
complications requiring the edit-build cycle.

And couldn't make-style this-has-already-been-done checking happen
with a copy anyway?

CHB

> Ah yes. So I don't think pip should do what it does. It a violation of
> the abstractions we all want to see within it. However its not me you
> need to convince ;).
>
> -Rob
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
> ___
> Distutils-SIG maillist  -  Distutils-SIG@python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] warning about potential problem for wheels

[Chris Barker - NOAA Federal]

But even if so do we WANT to now be supporting 4 Windows platforms?

win32
win32_sse2
win64
win64_sse2

and then, what about sse3, etc???

Longer term, more platforms (particularly more platforms that really only
differ by some compiler flags) shouldn’t be a very big deal, because my
dream for PyPI is that authors upload a source package and PyPI builds all
their wheels for them (optionally of course).

That would be pretty nice. I'm still a bit wary though -- the more
platforms you have, the more you have to test and debug

Testing, off course would be part of the automated build. But debugging
could get painful ...

CHB



-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] README.rst vs DESCRIPTION.rst

[Chris Barker - NOAA Federal]

Sent from my iPhone

On Sep 27, 2015, at 11:19 AM, Ionel Cristian Mărieș 
wrote:


On Sun, Sep 27, 2015 at 8:05 PM, Thomas Güttler 
​I don't think there can be a "definitive guide line"​. Unlike the core
language the packaging part of Python is a messy soup of different and
often competing ideas, styles and tools.


Which is EXACTLY why there should be one set of best-practices
recommendations that are the same in all the "official" docs.

I think one Readme.rst is the way to go.

If you want to provide contribution guidelines, etc. they should be in a
separate locations, referenced by the Readme.

-Chris



So you cannot have an definitive or objective guide for something that's
subjective in nature.

About the README vs DESCRIPTION - ask yourself, what would you use README
for then? I believe that's absolutely nothing. You only need one. :-)



Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro

___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Working toward Linux wheel support

[Chris Barker - NOAA Federal]

 On Jul 17, 2015, at 1:19 PM, Daniel Holth dho...@gmail.com wrote:

 I've recently packaged SDL2 for Windows as a wheel, without any Python code. 
 It is a conditional dependency if Windows for a SDL wrapper.

Cool, though I still think we need wheel-level deps -- the dependency
is on the particular binary, not the platform. But a good start.


 We were talking on this list about adding more categories to wheel, to make 
 it easier to install in abstract locations confdir, libdir etc. probably 
 per GNU convention which would map to /etc, /usr/share, and so forth based on 
 the platform.

Where would the concrete firs be? I think inside the Python install
I.e. Where everything is managed by python . I don't think I want pip
dumping stuff in /use/local, nevermind /usr. And presumably the goal
is to support virtualenv anyway.

 Someone needs to write that specification. Propose we forget about Windows 
 for the first revision, so that it is possible to get it done.

If we want Windows support in the long run -- and we do -- we should
be thinking about it from the start. But if it's going in the
Python-managed dirs, it doesn't have to follow Windows convention ...

 The real trick is when you have to depend on something that lives outside of 
 your packaging system, for example, it's probably easier to ship qt as a 
 wheel than to ship libc as a wheel.

Well, we can expect SOME base system! No system can exist without libc

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Making pip and PyPI work with conda packages

[Chris Barker - NOAA Federal]

Maybe I wasn't very clear -- I was addressing what conda might provide in
the context of using conda packages with pip/pipy.

A conda environment provides a great deal more, yes.

system-site-packages may very well be for a different version of the python
interpreter and stdlib.

Isn't that handled by the wheel meta-data? At least in practice -- Anaconda
is delivering python.org compatible pythons.

meta.yaml (w/ optional preprocessing # [selectors])
http://conda.pydata.org/docs/build.html calls build.sh or build.bat by
default, at build time.

Exactly -- you need to write those build scripts yourself. Conda does set
up the environment for you (lib paths,etc) so it's often as easy as
./configure  make  make install, but if it's not (particularly on
windows!) you've got to figure it out.

Which brings in the community aspect -- a lot of stuff has been figured out.

PPAs would be great

Personal package archives? Can't we do that now with pip+wheel? Indeed,
isn't that 1/2 of what binary wheels are for? And the only thing they are
for on Linux?

-Chris
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Chris Barker - NOAA Federal]

For the most part, I think it's all been said. What should and
shouldn't  be installed by default is really specific extension
dependent, not much point in speculating.

But a comment or two:


 having to type
 python -m something rather than just something isn't broken, it's
 just an inconvenience.

Tell that to a newbie. This is EXACTLY the kind of thing that should
just work.

Maybe a three-tier system:

1) mandatory -- can't install without it

2) default -- try to install it be default if possible

3) optional: only install if specifically asked for

And this isn't just about extensions -- for instance, the all stuff
in iPython would be well served by level 2

 It's no different conceptually than the fact that if you
 don't have a C compiler, you can't install a package that contains C

Sure it is -- a C complier is a system tool, and the whole point of
binary wheels is that the end user doesn't need one.

-CHB
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] setup_requires for dev environments

[Chris Barker - NOAA Federal]

Folks,

I'm having a hard time catching up with this, but maybe a few comments
from someone well outside the trenches will be helpful.

And note that one of use cases I'm now wrestling with is using another
package manager (conda), and setuptools is currently a PITA in that
context.

 you that python setup.py ... should
 install setup_requires? For me I'd be quite happy if installing the
 requirements was my own problem in the absence of an installer.

Yes, yes, yes! Separation of concerns is key here -- the tool that
builds and/or installs one package shouldn't do ANYTHING to get me
dependencies, except maybe warn that they are not there. And raise a
sensible error if a build dependency is not there.

Particularly for developers, they really are capable of installing dependencies.

 I've no particular thoughts on that. It would certainly avoid the pain
 of easy_install being triggered.

Ahh! Is that why this is so painful? Not only is setuptools trying to
install stuff for me, but it's using easy_install to do so? Aargh!

 Success criteria for my immediate personal needs:
 - pip install -e . works on a clean checkout of my projects

Sure.

 - easy_install doesn't go and download stuff

easy_install doesn't do anything, ever!

 - my setup.py can refer to things (usually the version) inside the
 project itself, safely

Yeah, that would be nice.

A few other notes:

If I have this right, this thread, and a number of other issues are
triggered by the fact that setup() is not declarative -- i.e. You
don't have access to the metadata until it's been run.

But maybe we can kludge a declarative interface I top of the existing
system. Something like requiring:

Setup_params = a_big_dict_of_stuff_to_pass_to_setup

setup(**a_big_dict_of_stuff_to_pass_to_setup)

 Code could look for that big dict before running setup. If it's not
there, you don't get any new functionality.

Note that I'm wary of a completely declarative system, there ends up
being a lot if stuff you don't want to hard-code, so you have to start
building up a whole macro-expansion system, etc. I'd much rather
simply let the user build up a python data structure however they want
-- the default, simple cases would still be basic declarative
hard-coding.

I suppose it's too late now, but the really painful parts of all this
seem to be due to overly aggressive backward compatibility. We now
have wheels, but also eggs, we now have pip, but also easy_install,
etc.

Perhaps it's time to restore distribute -- setuptools without the
cruft. Folks could choose to use distribute (or maybe setuptools2)
instead of setuptools, and not get the cruft.

pip would, of course, still need to work with setuptools, and
setuptools would have to be maintained, but it would give us a path
forward out of the mess.

Another issue I don't see a way out of is that the package name that
you use to ask for a package, say on pypi, is not necessarily the name
of the python package you can import. So it's really tricky to check
if a package is installed independently of the package manager at
hand. This is the source of my conda issues -- conda installs the
dependencies, but setuptools doesn't know that, so it tries to do it
again -- ouch.

Final note: setuptools has always bugged me, even though it provides
some great features. I think all my struggles with it come down to a
key issue: it does not make clear distinctions between what should
happen at build-time vs install-time vs run-time.

For example:

I don't want it downloading and installing dependencies when I go to
build. That's an install- time task.

I don't want it selecting versions at run time--also an install time task.

There are others I can't recall -- but a couple years back I was
bundling up an application with py2exe and py2app, and found I had to
put an enormous amount of cruft in to satisfy setuptools at run time
(including setuptools itself) -- it was pretty painful.

And of course, using it within another package manager, like conda --
I really want it to build, and only build, I'm taking care of
dependencies another way.

OK, I've had my rant!

-Chris
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wheels or system packages for pip on ubuntu

[Chris Barker - NOAA Federal]

Your might want to consider conda and conda environments for this.

http://www.continuum.io/blog/conda

It provides a single packaging solution for both python and
dependencies. And there are probably already recipes for everything
you need.

-Chris

 On Sep 3, 2014, at 3:24 AM, Reinout van Rees rein...@vanrees.org wrote:

 Hi,

 I'm investigating some options for making our servers a bit more neat. Basic 
 problem: lots of what we do needs mapnik, numpy, gdal, psycopg2 and so. 
 Python libraries with C code and system dependencies.

 All of them have ubuntu packages, but especially for gdal we sometimes need a 
 newer version. A PPA can help here, but I thought a wheel could be nice, 
 too.

 System packages? Yes, we use buildout with syseggrecipe. You pass 
 syseggrecipe a bunch of packages (mapnik, gdal), it looks up those packages 
 in the OS and installs them in buildout's develop-eggs/ directory. Works 
 quite well. Isolation + selective use of system packages.


 Two questions:

 a) If I use ubuntu packages, I'll have to run pip/virtualenv with 
 --system-site-packages. pip install numpy will find the global install just 
 fine. But pip freeze will give me all site packages' versions, which is not 
 what I want.

 = is there a way to *selectively* use such a system package in an 
 otherwise-isolated virtualenv?


 b) Making a bunch of wheels seems like a nice solution. Then you can just use 
 a virtualenv and pip install numpy gdal psycopg2  But how do you 
 differentiate between ubuntu versions? Not every wheel will work with both 
 12.04 and 14.04, I'd say. But both ubuntu versions will have the same 
 linux_x86_64 tag.

 = What is the best way to differentiate here? Separate company-internal 
 wheelhouse per ubuntu version? Custom tags?



 Reinout

 --
 Reinout van Rees  http://reinout.vanrees.org/
 rein...@vanrees.org   http://www.nelen-schuurmans.nl/
 Learning history by destroying artifacts is a time-honored atrocity

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 https://mail.python.org/mailman/listinfo/distutils-sig
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Handling the binary dependency management problem

[Chris Barker - NOAA Federal]

On Dec 5, 2013, at 1:40 AM, Paul Moore p.f.mo...@gmail.com wrote:


 I'm not sure how this would work - wheels don't seem to me to be
 appropriate for installing external dependencies, but as I'm not
 100% clear on what you mean by that term

One of the key features of conda is that it is not specifically tied
to python--it can manage any binary package for a system: this is a
key reason for it's existance -- continuum wants to support it's users
with one way to install all they stuff they need to do their work with
one cross-platform solution. This includes not just libraries that
python extensions require, but also non-python stuff like Fortran
compilers, other languages (like R), or who knows what?

As wheels and conda packages are both just archives, there's no reason
wheel couldn't grow that capability -- but I'm not at all sure we want
it to.

-Chris
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Handling the binary dependency management problem

[Chris Barker - NOAA Federal]

On Dec 4, 2013, at 11:35 PM, Ralf Gommers ralf.gomm...@gmail.com wrote



I'm just wondering how much we are making this hard for very little return.

I also don't know.


I wonder if a poll on the relevant lists would be helpful...


 I'll start playing with wheels in the near future.


Great! Thanks!

There are multiple ways to get a win64 install - Anaconda, EPD, WinPython,
Christoph's installers. So there's no big hurry here.


well, this discussion is about pip-installability, but yes, some of those
are python.org compatible: I know I always point people to Christoph's repo.



 [Side note: scipy really shouldn't be a monolithic package with everything
 and the kitchen sink in it -- this would all be a lot easier if it was a
 namespace package and people could get the non-Fortran stuff by
 itself...but I digress.]


Namespace packages have been tried with scikits - there's a reason why
scikit-learn and statsmodels spent a lot of effort dropping them. They
don't work. Scipy, while monolithic, works for users.


True--I've been trying out namespace packages for some far easier problems,
and you're right--not a robust solution.

That really should be fixed--but a whole new topic!




 Note on OS-X :  how long has it been since Apple shipped a 32 bit machine?
 Can we dump default 32 bit support? I'm pretty sure we don't need to do PPC
 anymore...


 I'd like to, but we decided to ship the exact same set of binaries as
 python.org - which means compiling on OS X 10.5/10.6 and including PPC +
 32-bit Intel.


 no it doesn't -- if we decide not to ship the 3.9, PPC + 32-bit Intel.
 binary -- why should that mean that we can't ship the Intel32+64 bit one?


But we do ship the 32+64-bit one (at least for Python 2.7 and 3.3). So
there shouldn't be any issue here.


Right--we just need the wheel. Which should be trivial for numpy on OS-X --
not the same sse issues.

Thanks for working on this.

- Chris
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Handling the binary dependency management problem

[Chris Barker - NOAA Federal]

On Dec 5, 2013, at 1:12 PM, Oscar Benjamin oscar.j.benja...@gmail.com wrote:


 Yes, I wouldn't want that kind of bad PR getting around about
 scientific Python Python is slower than Matlab etc.

Well, is that better or worse that 2% or less people finding they
can't run it on their old machines

 It seems as if there is a need to extend the pip+wheel+PyPI system
 before this can fully work for numpy.

Maybe, in this case, but with the whole fortran ABI thing, yes.

 You mentioned in another message that a post-install script seems best
 to you.

What would really be best is run-time selection of the appropriate lib
-- it would solve this problem, and allow users to re-distribute
working binaries via py2exe, etc. And not require opening a security
hole in wheels...

Not sure how hard that would be to do, though.

 3) Upload wheels for Windows to somewhere other than PyPI e.g.
 SourceForge pending a distribution solution that can detect SSE
 support on Windows.

The hard-core I want to use python instead of matlab users are being
re-directed to Anaconda or Canopy anyway. So maybe sub-optimal
binaries on pypi is OK.

By the way, anyone know what Anaconda and Canopy do about SSE and a good BLAS?


 I think it would be good to have a go at wheels even if they're not
 fully ready for PyPI (just in case some other issue surfaces in the
 process).

Absolutely!

- Chris
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Plans for binary wheels, and PyPi and OS-X

[Chris Barker - NOAA Federal]

On Nov 1, 2013, at 11:25 PM, Marcus Smith qwc...@gmail.com wrote:

 pip does confirm the wheel file is compatible with your platform/python 
 (based on the file tags), when downloading from indexes and links.
 BUT, just noticed, when installing directly from a specific file, it seems 
 it's not checking, so I'll look into that and log an issue

Great, thanks!
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Comments on PEP 426

[Chris Barker - NOAA Federal]

someone wrote:
  even though plain distutils would work just fine for a very
 large subset of packages.  Most of my own packages
 only use setuptools for its dependency resolution, and
 that's for the most part provided by pip in the future.


Another comment from the peanut gallery: This is very, very true -- I know
nothing of distutils internal design or extensibility, but from the user
end, it is really, really nice!

It's actually quite remarkable that it just works most of the time --
even building universal binaries on OS-X, and needing very little platform
dependent code in a setup.py.

So what problems are we really trying to solve?

For my part: I use setuptools for develop mode. That's it. Really.

Then there is pip and easy_install -- nice to have the auto-discovery,
download and dependency tracking -- though all of that is actually pretty
over-rated -- it makes it faster to install a complex, dependency-riddled
package (Like a web framework), but it breaks often enough that I end up
doing it by hand anyway. For me, pip install really only works well for
single, pure-python packages. (OK, some with external-dependency-free
extensions, but only because I compile my own stuff so have set up
the environment already -- and it's distutils that's working well there...)

On the other hand, setuptools introduced a lot of cruft that seems to get
in the way more than anything else.

- Package versioning is a great idea and _should_ be a standard part of
python, but the community seems to have settled on virtualenv as the
solution to that. And the setuptools solution never seemed to work for me.

- eggs: what's the point? and why mix a distribution format with an
installation, and binary with source, and zipped with unzipped?

- In general, setuptools seems to mix what should be install-time stuff
with run-time stuff -- makes it very hard to use things like py2exe,
py2app, etc. (pkg_resources is a mess, still not sure why it's needed)

OK: so what are problems that really do need to be solved?

Standard binaries that work with virtualenv -- as virtualenv seems to be
the solution to versioning, we really need to be able to either download
and install a binary in a virtualenv, or have pip do it for us -- wheels
and the proper naming schemes seems to be going in the right direction for
this.

The big one: ALL of the problems I have with building compiled stuff is due
to non-python external dependencies. Someone on the thread recently laid
out all the ugly issues with different compilers, etc on OS-X, Windows,
etc, etc. Where those all cause problems is the non-python dependencies:
libpng, libfreetype, libnetcdf, libhdf, etc, etc, etc. That's where it gets
ugly, and AFAIK, no one is trying to build a python solution (the only one
I know of is gattai, which I've started using and hacking on, l but it
really is just away to standardise your build scripts). Granted, I don't
know that it's possible or desirable to have a python solution for that --
that's what apt-get, rpm, homebrew, macports, etc, are for, but that's
where the real hang-ups are. Binary wheels _may_ help that a lot -- so very
few folks actually need to deal with it.

Sorry to sound critical here, but I'm hoping that with all this work, and
failed startes (distutils2) we won't just end up in the same
over-engineered crufty place we've been.

All that said -- I'm looking forward to wheels and a pip that understands
them -- IIUC, we have that, so it's on to the social part -- getting people
to test/use them!

I guess this reads like a rant, which is not really my intent -- it's great
to see all the work getting done, and people thinking hard about good
solutions. I really do appreciate all that effort.

-Chris


Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Multi-version import support for wheel files

[Chris Barker - NOAA Federal]

Just to add a bit more FUD ;-)


I do a lot of packaging things up with py2app, py2exe, etc. -- I find
I often want to be able to give folks one thing that they can
install and run, and I'd rather they don't even need to know it's
built with python.

A while back, when I was doing this with a web app (actually a
Browser Interface, Local Server (BILS) app --i.e, a web server and
html  interface, but the browser embedded in a wx app so it acted like
a desktop app to the user...) setuptools, and pkg_resources really
made it hard.

I don't remember the details at this point, but it was built on
Pylons, which made heavy use of setuptools at the time. And
easy_install pylons worked great. But bundling it up was a pain -- we
had to put an enormous amount of crap in there to satisfy
pkg_resources, and who knows what -- crap that was not used, and
should not have been needed at run time. In fact, we needed to put in
at least the skeleton of stuff that was a different version than the
one used.

Note also that the only reliable way to get an egg-installed package
to work with py2app was to include the whole egg.

My conclusion at the time was that there was a real confusion between
what should happen at install time, and what should happen at run
time. For instance, making sure that a dependency is there and the
right version seems like an install-time problem to me -- and yet,
there is was, getting checked every time you started up the app. I'm
not clear at this point how much of that was inherent to how
setuptools works, and how much was how Pylons chose to use it, but I
hope we'll all keep this use-case in mind at this point.

i.e. if a user has only needs one version of a given package
installed, lets not have much overhead there to support that, and
let's not require much run-time support at all.

I note that I've been a user of wx.version for wxPython, and it was
always easy and painless, including py2app, py2exe use. So it can be
done.

-Chris


On Mon, Aug 26, 2013 at 8:15 AM, Donald Stufft don...@stufft.io wrote:

 On Aug 26, 2013, at 10:33 AM, PJ Eby p...@telecommunity.com wrote:

 On Mon, Aug 26, 2013 at 5:20 AM, Paul Moore p.f.mo...@gmail.com wrote:
 On 25 August 2013 23:14, PJ Eby p...@telecommunity.com wrote:
 Thus, you don't have to know you have multiple versions installed; it
 can trivially happen by way of dependencies you aren't paying
 attention to.  The more things you install, the more likely it is you
 have two versions hanging around.


 OK, I see. But I'm not sure if we're agreeing or disagreeing over the
 result. To me, this is a bad thing on the principle that there is a cost to
 multiversion support (it's not part of core Python, so you have to do
 *something* to make it work)

 Seriously?  The basic functionality of using sys.path to have multiple
 versions *is* part of core Python, and has been since 1.5.2 (16 years
 ago), and probably longer than that.

 In the days before easy_install and virtualenv, if you needed
 different versions of things, you used setup.py install to different
 directories (assuming distutils was involved, otherwise you just
 copied files) and either put your scripts in the same directories, or
 used PYTHONPATH or explicit sys.path manipulation.

 That is all easy_install does: add a naming convention for the
 directories, and automate the sys.path manipulation.

 Buildout does the same thing, it just writes the sys.path manipulation
 into the scripts statically, instead of using pkg_resources at
 runtime.

 So the notion of cost doesn't make any sense.  Tools like
 easy_install and buildout *reduce* the management cost, they don't add
 anything to core Python.

 (Now, if you're talking about the .pth files from easy_install, those
 are something that I added because people complained about having to
 use require(), and wanted to have a default version available in the
 interpreter.)


 and so having people inadvertently pay that
 cost to use a feature that they don't actually *need* is wrong.

 What cost are you talking about here?  Given that most people don't
 even know they *have* multiple versions installed or care, how is a
 cost being imposed upon them?  Are you talking about disk storage?


 One other point, just as a matter of curiosity (because it's not relevant to
 the current discussion): in your explanation above, there doesn't seem to be
 any step that says the user normally uses CherryPy 3 (so that would be the
 one they would get automatically at the interactive interpreter).

 If they easy_install that version, sure, that's what they'll get as a
 default version.


 For me,
 that's really the only use case I'd have for multi-versioning - 99% of the
 time I use a particular version of a project, but I have one particular
 application that can't work with the version I prefer.

 Yes, and that's the sort of scenario Nick was proposing pip support,
 that you have an explicit install me a different version for my other
 app capability -- such that 

Re: [Distutils] Multi-version import support for wheel files

[Chris Barker - NOAA Federal]

PJE:

Thanks for the clarification: based on that: +1 on Nick's proposal.

Chris

On Aug 26, 2013, at 1:41 PM, PJ Eby p...@telecommunity.com wrote:

 On Mon, Aug 26, 2013 at 11:15 AM, Donald Stufft don...@stufft.io wrote:
 There is always a cost. In this case mostly in complexity and start up time.

 As you mentioned originally the cost to multi version support was the need
 to use a require() function and when people complained about that you
 added the .pth files which imposed another sort of cost to people using
 multi versioned installs.

 See, this is exactly what I'm talking about: you've got this 100% backwards:

 .pth files are for people who *aren't* using multi-version imports.
 They're for *default* versions, not alternate versions!

 And they're utterly unnecessary for Nick's proposal.


 You claim it is part of core Python but it's really not, if it was it 
 wouldn't require
 importing pkg_resources of the .pth files to make it work.

 As I pointed out in the email you apparently didn't read, along with
 multiple emails from Jim: pkg_resources isn't necessary for
 alternate-version support.  All that's required for alternate versions
 is to add them to sys.path, which buildout does just fine *without
 pkg_resources*.


 I find it ridiculous that you'd call this thread 90% FUD when the vast bulk 
 of the
 thread has been trying to determine if there were any reasonable concerns
 with the approach and upon examination determined that the biggest problem
 with it was attaching it to Wheel and not the multi version support at all

 What I'm referring to as the FUD is that people have been confusing
 what Nick proposed with what setuptools does, and getting *both* of
 them wrong in the details.

 Nick's proposal was not to mimic setuptools' multi-version support,
 but rather to provide something else: let's call it alternate version
 support, to separate it from what setuptools does.

 In Nick's AVS proposal, there is *no* overhead for anything that
 doesn't need a non-default version, and it's 100% opt-in, used only
 for things that need *non-default* versions.

 Note, by the way, that since these *non-default* packages aren't on
 sys.path by default, *there is no overhead and no .pth files are
 involved*.  They are effectively invisible and irrelevant for anything
 that doesn't use them.

 The only place where there's overhead is in the script that needs the
 alternative version(s), and its sys.path is lengthened only by those
 items that it can't obtain from the default sys.path.  And if you use
 buildout's approach of simply adding:

 sys.path[0:0] = [path1,...]

 to the head of a script, then *pkg_resources isn't involved either*.

 This is bog-standard stock Python.

 So the FUD part I was referring to is all the oh no, setuptools is
 complicated in response to Nick's perfectly reasonable idea *which
 doesn't involve any of setuptools' complexity*, because it's doing
 something completely different.


 I realize
 setuptools and easy_install are your baby but the persecution complex doesn't
 help to win people over to your side of things.

 I think you're confused here.  I don't think setuptools is being
 persecuted, I think *Nick's idea* is being misunderstood, and being
 construed as almost the exact *opposite* of what it is.

 All the stuff people bitch about that relates to multi-versions in
 setuptools are actually issues with setuptools' implementation of
 *default* versions, not *alternative* versions.  So to look at Nick's
 proposal and think it's going to have the same problems is completely
 ludicrous - it's 180 degrees opposite of what setuptools does, because
 for setuptools, *default versions* are the special case -- they're
 what cause 90% of the complexity in pkg_resources' manipulation of
 sys.path, and they're the main reason .pth files are ever used.

 So it's crazy-making to see people thinking Nick's proposal is going
 to bring all that crap along, when that's the exact *opposite* of the
 situation.


 In my experience setuptools has a lot of good ideas but they are wrapped in 
 bad
 ideas or implementations that obscure the fact that there *are* good ideas 
 there.
 I do not believe it to be unreasonable for people to want to make sure that 
 we're
 standardizing around one of the *good* ideas instead of one of the bad ideas.

 It would help if people understood the actual facts, then.  AFAICT,
 Nick's proposal doesn't do any of the things that people are worried
 about, or at the very least does not *require* them.  As Jim and I
 have pointed out more than once, pkg_resources is not a runtime
 requirement to allow alternative versions to be importable by code
 that wants them.

 It would really be a shame to shoot down Nick's idea based on a vague
 misunderstanding of it.  It's a good proposal, and has far less to do
 with setuptools than most people in the thread seem to think.
 ___
 Distutils-SIG maillist  -  

Re: [Distutils] What does it mean for Python to bundle pip?

[Chris Barker - NOAA Federal]

On Thu, Aug 22, 2013 at 6:52 AM, Oscar Benjamin
oscar.j.benja...@gmail.com wrote:

 I'm pretty sure the current Windows installer just doesn't bother with
 BLAS/LAPACK libraries. Maybe it will become possible to expose them
 via a separate wheel-distributed PyPI name one day.

Well, the rule of thumb with Windows binaries is that you bundle in
(usually via static linking) all the libs you need -- numpy could have
a semi-optimized LAPACK or not, and the user shouldn't know the
difference at install time. But the trick in this case is that numpy
is used by itself, but also widely used with external C and Fortran
that might want LAPACK. (including scipy, in fact...)

But maybe this is all too much to bite off for pip and wheels. If we
could get to a state where pip install numpy and pip install scipy
would do something reasonable, if not optimized, I think that would be
great! And it's really not a big deal to say:

If you want an optimized LAPACK, etc.  for your system, you need to do
something special/ by hand/ etc...

something special may be as simple as download
numpy_optimized_just_for_this_machine.whl and install it with pip.

All that being said -- OS-X has a moderately complex binary set, what
with universal binaries, so maybe we can have a bit more meta-data
about the architecture supported.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

[Chris Barker - NOAA Federal]

On Thu, Aug 22, 2013 at 9:37 AM, Paul Moore p.f.mo...@gmail.com wrote:

 That is essentially possible now.

 1. Go to Christoph Gohlke's website and download his bdist_wininst
 installers for numpy and scipy.



Exactly. And when all this settles down, hopefully Christoph, and
others, will put up binary wheels and we'll have one stop installing
that supports virtualenv, and pypi discover of good enough binary
wheels.

My point is that is may not be wise to try to support the more complex
builds -- they ARE complex, and trying to suppor it with an auto tool
is a bit much.

Oscar wrote:
 And actually 'pip wheel numpy' works fine for me on Windows with MinGW
installed.

Good start, but the bigger issue is that 'pip install' finds that wheel...

I'm still confused as to the state of all this -- are the tools ready
for project to start posting wheels so that pip can find them?

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

[Chris Barker - NOAA Federal]

On Thu, Aug 22, 2013 at 3:52 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 22 August 2013 23:08, Chris Barker - NOAA Federal chris.bar...@noaa.gov

 My impression is that the architecture and fat binary stuff on OSX is the
 bit that may bite you.

exactly.

 I know little or nothing about OSX, but I'm sure if
 you try and report on how you get on, the people on the list will be able to
 help you get things sorted and we will be able to get any dark corners
 ironed out.

yup -- we'll never figure it out if no one uses it...

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] What does it mean for Python to bundle pip?

[Chris Barker - NOAA Federal]

On Thu, Aug 22, 2013 at 8:17 PM, Nick Coghlan ncogh...@gmail.com wrote:

 numpy-1.7.1-cp27-cp22m-win32.whl
 numpy-1.7.1-cp27-cp22m-win32-sse.whl
 numpy-1.7.1-cp27-cp22m-win32-sse2.whl
 numpy-1.7.1-cp27-cp22m-win32-sse3.whl

I'm still confused -- how would pip install numpy know which of
these to install?

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] distutils.util.get_platform() - Linux vs Windows

[Chris Barker - NOAA Federal]

On Tue, Aug 20, 2013 at 9:00 AM,  samuel.feren...@barclays.com wrote:

 That's strange. I'm on Python 3.3.1, and it seems to me that get_platform()
 derives the value from uname for OS X, similar to Linux.

 (osname, host, release, version, machine) = os.uname()
 ...
 elif osname[:6] == darwin:
 import _osx_support, distutils.sysconfig
 osname, release, machine = _osx_support.get_platform_osx(
 distutils.sysconfig.get_config_vars(),
 osname, release, machine)
 return %s-%s-%s % (osname, release, machine)

 so I would expect uname -m to be in line with get_plaform(). But maybe I'm
 misreading that... Also, I don't have access to the _osx_support source code.

yup -- _osx_support.get_platform_osx() does special magic

 return value is wrong on Linux and correct on
 Windows, right?

 no -- I'm saying that it's right on Windows (and OS-X), but wrong on Linux.

 I think you have misread my sentence, and we actually agree here.

duh! yes, we do.

 What's the next action? Report a Python bug? (That's a cultural question; I'm
 new to Python.)

not sure -- this seems like the right place to report it, but an
offical bug report may make sense.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] distutils.util.get_platform() - Linux vs Windows

[Chris Barker - NOAA Federal]

On Mon, Aug 19, 2013 at 11:15 PM,  samuel.feren...@barclays.com wrote:

 What does your 'uname -m' return?

x86_64

 Is it possible you're really running a 32-bit
 Python on a *32-bit* OS X kernel? [http://superuser.com/q/161195]

nope -- I am quite deliberately running a 32 bit Python on my 64 bit
OS (I have some custom code C++ Im using that is not yet 64 bit
safe).

 return value is wrong on Linux and correct on
 Windows, right?

no -- I'm saying that it's right on Windows (and OS-X), but wrong on Linux.

That get_platform() should return 32-bit for a 32-bit process
 running on a 64-bit system.

yes, it should.

 TBH, I was expecting the opposite; to me, platform
 means the OS, which would mean that Linux does well to derive the return value
 from the OS's architecture.

except what would be the utility of that? this is a call made within
python, and it's part of distutils, so what the caller wants to know
is the platform that this particular python was build for, NOT the
platform is happens to be running on. i.e. what platform do I want to
build binary extensions for, and/or what platform do I want to
download binary wheels for.

So I'm pretty sure that currently Windows and OS-X have it right, and
Linux is broken. I'm guessing running 32 bit python on a 64 bit LInux
is not that common, however. (and it's less common to download
binaries...)

To add complexity, if I run  the Apple-supplied python2.7.1 (which is
32_64 bit universal, but runs 64 bit on my machine), I get:

 distutils.util.get_platform()
'macosx-10.7-intel'

Which is more useful than it may look at first -- intel means both
intel platforms, i.e. i386 and x86_64. and 10.7 means -- built for
OS-X 10.7 and above.

so I think it's doing the right thing.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to handle launcher script importability?

[Chris Barker - NOAA Federal]

Just $0.02 from a user...

I'm primarily an OS-X user these days, but have to do Windows once in
a while, and help others do Windows (including as an intro to Python
instructor)

Once I discovered setuptools develop mode, I never looked bak -- it
is simpl;y THE way to develop code, particularly if you are working on
a lib you want to use from other projects and/or actively develping
utility scripts.

I really like that I can add some scripts to setup.py, and use either
develop mode or regular old install and get nice commands, and it
works the same way on Windows and *nix (with the hood closed0

I did all this for a good while before I even noticed that exe
launchers  -- it just worked. In fact, the only time I noticed the
launchers was a couple years ago when a beta version of setuptools
released a broken version -- very frustration -- it would fire up
another command window that would then close when the script was done
-- not very helpful for nosetests and the like... When I did discover
how it all worked, I did think it was a little weird, but Windows
simply hasn't been built for command line stuff, so you do what you
have to do.

Conclusions:

1) an extra bunch of files is a on-issue for most users -- we just
need something that works.

2) the exe launcher is a bit fragile and hard to maintain (and even
harder to debug)  -- but there are smart people working on this.

3) I'd rather not have to mess with PATHEXT, and I particularly don't
want to have to tell my students to do it -- environment variables are
a pain, and somehow PATHEXT has been fragile for me (and I don't use
Cygwin)

I cant help thinking a more elegant solution exists, but maybe not.

Thanks to everyone hashing this out!

-Chris








On Tue, Aug 13, 2013 at 12:58 PM, Paul Moore p.f.mo...@gmail.com wrote:
 On 13 August 2013 18:08, Oscar Benjamin oscar.j.benja...@gmail.com wrote:

 On 13 August 2013 17:33, Paul Moore p.f.mo...@gmail.com wrote:
 
  On another point you mention, Cygwin Python should be using Unix-style
  shell
  script wrappers, not Windows-style exes, surely? The whole point of
  Cygwin
  is that it emulates Unix, after all... So I don't see that as an
  argument
  either way.

 So say I have a ~/bin directory where I put my scripts that I want to
 be generally available. I install something with
 python setup.py install --install-scripts=~/bin
 so that the scripts/script-wrappers go in there because I want to be
 able to always access that program under that name. Don't be fooled by
 the unixy tilde: I'm running ordinary Windows Python in that command
 in git-bash, not Cygwin. Now if that folder is on PATH while I am in
 Cygwin I can run the program with the same name if an .exe wrapper was
 added. I can't run it with the same name if it's a .py/,bat file
 because Cygwin doesn't have the implicit strip-the-extension PATHEXT
 feature and can't run .bat files.


 Ah, OK, thanks for the clarification.

 In that case I can see why you'd prefer exe wrappers (or maybe cygwin bash
 shell wrappers, or shell aliases...). Maybe an option to still use exe
 wrappers is worth it - but honestly, I'd say that in that context you
 probably have enough expertise to understand the issue and make your own
 solution relatively easily.

 What about having in your .bashrc:

 for prog in ls ~/bin/*.py; do
 alias $(basename $prog .py)=$prog
 done

 (Excuse me if I got the precise details wrong there). OK, you need to rerun
 .bashrc if you add new scripts. It's not perfect. But it's not a showstopper
 either.

 I do think, as I said before, that this needs some sort of policy-type PEP
 on the standard approach for wrapping scripts, with all the pros and cons of
 the various approaches documented and reviewed.

 Paul

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] How to handle launcher script importability?

[Chris Barker - NOAA Federal]

On Tue, Aug 13, 2013 at 2:27 PM, Paul Moore p.f.mo...@gmail.com wrote:

 3) I'd rather not have to mess with PATHEXT, and I particularly don't
 want to have to tell my students to do it -- environment variables are
 a pain, and somehow PATHEXT has been fragile for me (and I don't use
 Cygwin)

 Nobody is suggesting that end users mess with PATHEXT. The proposal is that
 the Python installer does this

ouch! I don't think I'd want PATHEXT set for *.py files -- I'd rather
they get opened by an editor by default than run...or is point+click
behavior different than command line -- shows you how well I know
Windows.

 (indeed, that's been done for Python 3.4, I
 don't know if the installer for the standalone launcher has been updated in
 the same way yet).

There are those of us still in the 2.7 world -- and I suspect for a good while.

 I'm suggesting that we collect specifics on any fragility (can you provide
 details of what has gone wrong for you?)

well, for PATHEXT, the env variable has to be set right, and Windows
kind of hides all that from you -- it's really a pain to edit them by
hand, so if the installer doesn't do it, or someone re-builds their
registry or profile, or what have you, then it'll break. Oh and the
cygwin (and who know what other shell alternatives) issue. At least we
can pretty much count on an exe running if the shell can find it...

Fragiity for the exe approach -- all I know is that the setuptools
binary was proken a while back -- but setuptools itself was in a bit
of a void of not-quite-sure-if-its maintained, and
not-sure-even-how-to-report-a-bug state. It seems that setuptools (or
whatever this will be part of) has now been adopted by the core Python
community, so we can expect good support -- let's hope so. Anyway,
whether the exe approach was more or less fragile than anyting else,
is sure was harder to debug/fix for anyone that doesn't know Windows
and non-python development well.


 I cant help thinking a more elegant solution exists, but maybe not.

 Personally, I believe that executable .py scripts (or maybe a dedicated
 .pys/.pye/.pya/whatever extension) *is* a more elegant solution - but
 equally, I concede, maybe not...

well, I guess that's the way Windows does things -- *nix has the
executable bit, Windows uses extensions, so I suppose that is the
way to get an executable script -- but it really SHOULDN'T be *.py!

 I think it's worth trying, though.

agreed.

I've lost track of what needs to be tried -- can't we just associate
an extension with py and give it a go? (got to get that Windos VM
working)

 Also, you mention develop mode. I don't use develop mode, most of my
 standalone scripts are single-file scripts, not anything that I'd bother
 packaging up with a setup.py, etc.

Good point -- it seem most of my scripts are part of a larger package,
either a set of scripts, or a couple scripts that all rely on a
particular package, so the whole setup.py thing works well.

but for a single stand-alone script, the PATHEXT and py launcher
approach seems really natural.

 And many of the issues I've had with the exe wrappers are particular to
 built installers (wheels, wininsts) and *not* develop mode.

in theory, wheels and develop mode should do the same thing -- not
sure about wininsts -- previously, the launcher thing was setuptools,
not plain distutils, not sure how that was handled.

but it would be best if they all did it the same way.

 PS I still think that long-term a policy PEP on the recommended way of
 making executable scripts is worthwhile.

Probably, yes:  There should be one-- and preferably only one
--obvious way to do it.

I'm leaning toward the PATHEXT approach -- it seems more si milar to
the *nix way, and perhaps easeir for lay folks to debug and fix. But
I'm saying tha tthe exe method worked fine for many of us, too.




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Expectations on how pip needs to change for Python 3.4

[Chris Barker - NOAA Federal]

On Sat, Jul 13, 2013 at 12:59 PM, Paul Moore p.f.mo...@gmail.com wrote:
 2. This sounds like something that needs fixed on Windows. Even if you say
 ``-m`` for pip then things are still broken by default for any other package
 on PyPI that installs a script. So this feels like something wrong with
 Python on windows not wrong with the script approach.

 It is, and it should be fixed. But in many years, nobody has managed to come
 up with an acceptable solution.

I don't _think_ this is just Windows Bashing: MS has done very very
little to improve the whole command line experience on Windows over
the years. For example, as far as I now, even with Windows 7 (8), the
standard system tool to edit PATH is a very, very old little text box
that only holds maybe 50 characters -- it's really painful and
pathetic. That, and I think the really, really old way of editing
autoexec.bat is dead (editing a text file is easier than a really
lousy GUI)

All that is a way to say that Python can only make it so easy for
Windows users, but what's in place is not bad, and it really makes
sense for pip to use what's been there for ages, i.e. a command called
pip (and pip2, pip3...) that sits in the same place that all other
third-party Pyton scripts are installed. No matter how you slice it,
a user will need to put that on their PATH one way or another.

Of course, what MS is telling us is: don't rely on the command line!
So a really nice thing to do for Windows users would be to provide a
little GUI pip tool that's part of the standard install. (not that I'm
volunteering to write it...has no none yet written a tkInter-bsed pip
front-end?)

the current setuptools exe-wrapper feels really kludgy, but it works
-- it seems the only real problematic issue is the self-update problem
-- maybe there is a Windows guru somewhere that can fix that

 The debates seem to be largely around what
 happens if you install multiple versions of Python and then remove some of
 them, and how badly your system PATH gets messed up by this.

Is this any better on *nix? When I use the OS-X installers, after a
while, I get a pretty klunky pile-up of PATH-manipulating stuf in my
.bash_profile...

 * Accept that Windows is a problem in this regard, but don't worry about it
 - install executable wrappers/scripts and let the user deal with path
 issues.

not so bad, really

 It would be nice to get feedback from normal users on this. I suspect that
 the scientific community would make a good cross-section (AIUI there's quite
 a lot of Windows use, and for many people in the community Python is very
 much a tool, rather than a way of life :-)).

True -- note that there are now two commercial pyton distributions
(Enthought Canopy, and Continuum Anaconda) that heavily used by the
scipy community -- they both provide their own package distribution
solutions (though ship pip, too, i'm pretty sure). The demand for
those tells us something about packaging

 Does anyone have links into the
 scipy groups?

Yes, but I don't know that post from me would get you anything that
that post from a core pip-developer wouldn't get -- I'd post on the
numpy list for best access to developers, maybe scipy and/or
matplotlib for more it's-just-a-tool-to-me users. iPython's not a bad
option for folks concerned about user experience, as well.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Expectations on how pip needs to change for Python 3.4

[Chris Barker - NOAA Federal]

On Mon, Jul 15, 2013 at 10:11 AM, Paul Moore p.f.mo...@gmail.com wrote:
 of Steve Dower, I guess :-)) Powershell is a *great* step up from cmd,

could we use a powershell script to launch python scripts? Maybe it
wouldn't be any easier to update than an exe, but it might be more
accessible.

 Of course, what MS is telling us is: don't rely on the command line!
 So a really nice thing to do for Windows users would be to provide a
 little GUI pip tool that's part of the standard install. (not that I'm
 volunteering to write it...has no none yet written a tkInter-bsed pip
 front-end?)

 I don't think a GUI-based tool is the answer here - the command line is
 orders of magnitude more powerful. For simple cases yes, but we have
 bdist_wininst and bdist_msi for those, and they are clearly not enough.

but they are really widely used -- maybe when binary wheels become
ubiqitous, I'll stop using them, but I'm no command line phobic, and I
usually go first to look for an installer on Windows. Over in the Mac
world, we have similar issues (except a proper command line under
there if you want it...), and eggs were a real issue because there was
nothing to launch if you point and clicked on one.

That being said, you're only going to get so far programmin python if
you can't run a simple command on the command line. So maybe any GUI
front-end should be part of a larger tool --  perhaps provided by IDE
developers, for instance.


 Most
 of my problems with the setuptools wrappers are not actually with the exe,
 but rather with the actual script (and its dependency on pkg_resources) that
 lies behind it - and that's not a Windows problem per se.

I've never liked pkg_resources. ;-)

 The real problem is not technical, actually - it's knowing what Windows
 users will actually be comfortable with. Unix users tend to assume Windows
 users are very uncomfortable on the command line (no offence meant to anyone
 by that) whereas the reality is that some are, some (like me...) really are
 not, and some are simply unfamiliar with the capabilities of the Windows
 command line through lack of need to use it (many of my colleagues, for
 example).

true -- and a simple command line solution is fine for most -- as I
said, they'll need to deal with that one way or another if they are
going to program..(and I say this as an instructor of intro to pyton
classes...)

 I'm actually tempted to give up trying to please everyone, and just put
 together a solution that suits *me* and see how that flies. Second guessing
 what other people want makes my head hurt :-)

fine plan!

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Expectations on how pip needs to change for Python 3.4

[Chris Barker - NOAA Federal]

On Mon, Jul 15, 2013 at 3:28 PM, Donald Stufft don...@stufft.io wrote:
 There is something like 200 total bdist_msi on PyPI and 5k bdist_wininst.

 To put numbers into perspective, there are ~180k total files uploaded to
 PyPI.

I don't hink this means that the installers aren't widely used, I
think it mean they aren't distributed on PyPI.

Installers are really useful for packages that require compiled code
that depends on external libs -- and most of the major such package
maintainers provide them.

Also, numbers aren't as important as the handful of widely used, but
hard to build, packages

But they are useless with virtualenv, so I'm looking forward to binary wheels...

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] $MACOSX_DEPLOYMENT_TARGET mismatch

[Chris Barker - NOAA Federal]

This is really a build question, rather than a distributuion question
-- Id try the pythonmac list:

http://mail.python.org/mailman/listinfo/pythonmac-sig

I recall that readline is a bit of a pain on the Mac, but don't recall
the solution (nor am I running 10.8 yet).

Good luck,
  -Chris

On Wed, Jul 3, 2013 at 6:55 AM, Alan alanwil...@gmail.com wrote:
 Hi there,

 I am trying to install readline and cx_Oracle on my Mac OSX 10.8.4. I've
 just installed python 2.7.3 standard in my $HOME dir and then I did:

 export PATH=$HOME/bin:$PATH
 ~/setuptools-0.7.7
 python setup.py install --prefix=$HOME

 which worked.

 I tried to installed ipython 0.13.2 which seems to have worked but it fails
 to start because of readline.

 easy_install readline
 Searching for readline
 Reading https://pypi.python.org/simple/readline/
 Best match: readline 6.2.4.1
 Downloading
 https://pypi.python.org/packages/source/r/readline/readline-6.2.4.1.tar.gz#md5=578237939c81fdbc2c8334d168b17907
 Processing readline-6.2.4.1.tar.gz
 Writing
 /var/folders/0c/n_0by3qj74bg5m68fxrw2kw4gp/T/easy_install-3WQA6K/readline-6.2.4.1/setup.cfg
 Running readline-6.2.4.1/setup.py -q bdist_egg --dist-dir
 /var/folders/0c/n_0by3qj74bg5m68fxrw2kw4gp/T/easy_install-3WQA6K/readline-6.2.4.1/egg-dist-tmp-eyGcUE
 error: Setup script exited with error: $MACOSX_DEPLOYMENT_TARGET mismatch:
 now 10.3 but 10.4 during configure

 How can solve this problem please?

 Thanks in advance,

 Alan

 --
 Alan Wilter SOUSA da SILVA, DSc
 Bioinformatician, UniProt - PANDA, EMBL-EBI
 CB10 1SD, Hinxton, Cambridge, UK
 +44 1223 49 4588

 ___
 Distutils-SIG maillist  -  Distutils-SIG@python.org
 http://mail.python.org/mailman/listinfo/distutils-sig




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] $MACOSX_DEPLOYMENT_TARGET mismatch

[Chris Barker - NOAA Federal]

On Wed, Jul 3, 2013 at 12:12 PM, Alan alanwil...@gmail.com wrote:
 Well, I found out that if before compiling my python I set

 export MACOSX_DEPLOYMENT_TARGET=10.3

 and then do all the rest, then I get easy_install to work.

cool - well done.

 So, somehow my python (or setuptools) need to build for 10.3.

 For me it's a bug in setuptools since the function that do this check and
 raise the error are from setuptools package.

 I've just installed python 2.7.3 standard

How did you install it? not sure what standard is. The build is
supposed to setup distutils to do the right thing, and I'm pretty sure
it does with the binaries from python.org.

I will note that unless you want to re-package (i.e. with py2app) and
distribute to folks with old machines, you probably don't need to
support way back to 10.3 -- the intel build on python.org is set up
for 10.5 and greater, and may avoid this issue.

-Chris

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] wxPython and wx

[Chris Barker - NOAA Federal]

On Fri, Jun 7, 2013 at 12:56 AM, Yuval Greenfield ubershme...@gmail.com wrote:

 I realize we don't want to start squat-wars. But pypi's wx is a 5 liner that
 absolutely no person would ever want installed. At the least, pip should
 warn me when I install wx that it's probably not what I'm looking for.

This isn't a question for the wxPython project, it's one for the PyPi
folks and/or the author of the other wx.

As it happens, there has been a discussion recently on the distutils
list about whether the Pyton community should establish a system for
regulating top-level package names -- but at this point, it seems
everyone all getting along is where we are at.

wxpython is listed on PyPi, what else can we do? This is not a unique
situation that someone would accidentally install a different package
than they intended. In this case, I suspect that if you installed
wxPython on top of that wx, it would just work.

Still not a bad idea to suggest the author of the other wx to remove
or rename it -- why invite confusion?

Last point -- does PyPi have a system for aliases? If so, it might be
a good idea to have wxPython in there as wx as well.

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Binary Wheels and universal builds on OS-X

[Chris Barker - NOAA Federal]

On Tue, Jun 4, 2013 at 1:23 AM, Ronald Oussoren ronaldousso...@mac.com wrote:

 This isn't really a problem, distutils uses labels for the set of supported 
 architectures as the architecture label in binary distributions (e.g. 
 pyobjc_core-2.5-py3.4-macosx-10.8-intel.egg for an egg that supports the 
 'intel' set of architectures (x86_64 and i386), see 
 _osx_support.get_platform_osx in the CPython repository for the labels that 
 are currently used.

 This is not ideal, but works good enough for now. In the long run this should 
 likely be replaced by the compressed tags sets from PEP 425 (at the cost of a 
 much longer file names).

I think now may be the long run. But in any case, as we were
discussing, as in theory there could be lots of possible combinations,
in practice there are only two in the wild (plus the non-universal
ones...), and probably only one we really  need to keep supporting, so
simply have a canonical name for the one or two is fine.

 A much more interesting question is which binary wheels should be considered 
 when installing a package,

indeed.

 there a two clear ways to select binaries and both have there usecases. Let's 
 say you're running on OSX 10.8 with a CPU that supports x86_64 and a Python 
 installation with support for x86_64, i386 and ppc and compiled with support 
 for OSX 10.4 or later (MACOSX_DEPLOYMENT_TARGET=10.4).  When installing a 
 package you can pick either:

 1) a binary wheel that supports this machine (a wheel compiled with 
 deployment target 10.8 with only x86_64 support)

can you link a newer deployment target-built lib with an older
deployment-target executable? I guess so...

 2) a binary wheel that supports the same systems as the Python installations

 The former is good enough when you only want to run code on your machine, the 
 latter can be needed when you want to deploy the installed binaries to 
 another machine (for example because you are using py2app to create an 
 application bundle that will be installed elsewhere).

I would say that we should clearly _prefer_ a wheel that fully
supports the pyton the installer is used with.

But I'm a bit ambivalent about what to do if there is no such wheel
found, but there is one that supports the currently running
architecture. Im inclined to say don't install it:

 - We build the universal binaries to support people that don't want
to know or care about such details.

- We want to support universal binary wheels for those folks.

- The primary reason to build binary wheels at all is to support
non-native architectures -- it's a whole lot easier to simply build
for your current machine, and homebrew and macports, and ??? support
that already.

All that being said, I sometimes think that we should simply have a 32
bit and 64 bit binary out there, and forget all this universal stuff!
It works OK for Windows

 On my system right now, I have 32bitPPC + 32bit i386 universal python,
 running on an intel system:

 In [2]: distutils.util.get_platform()
 Out[2]: 'macosx-10.3-i386'

 On my machine this returns the correct value: 'macosx-10.8-intel' (binary 
 supports 10.8 or later and the i386 and x86_64 architectures). I do know this 
 used to return wrong values on some version of OSX when using the Apple 
 installation of Python because of the way they build Python.

This isn't an Apple build -- I'm pretty sure it's the one from
Python.org...though looking now it's only 2.7.2 -- I guess I haven't
upgraded in a while!

now with 2.7.5:

In [3]: distutils.util.get_platform()
Out[3]: 'macosx-10.3-fat'

I guess fat means 32bit ppc + intel, deployment target 10.3 ?

So we may want to update distutils.util.get_platform to do a PEP 425
name, but we've got something that could work now.

-Chris






-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Binary Wheels and universal builds on OS-X

[Chris Barker - NOAA Federal]

On Mon, Jun 3, 2013 at 3:47 PM, Daniel Holth dho...@gmail.com wrote:

 That would make sense. Can you come up with code to detect that a
 newly compiled extension is universal, and that a Python is?

It looks like distutils.util.get_platform() now does the right thing
for knowing what the currently running pyton is (see Ronald's message)

For determining the status of a newly compiled extension, I usually
simple run the file command line utility on it:

$ file python
python: Mach-O universal binary with 2 architectures
python (for architecture ppc):  Mach-O executable ppc
python (for architecture i386): Mach-O executable i386

(works for *.so, too...)

that could easily be parsed out, but we would still want to know the
deployment target, which maybe you could get by parsing otool output:

$ otool -L python
python (architecture ppc):
/Library/Frameworks/Python.framework/Versions/2.7/Python
(compatibility version 2.7.0, current version 2.7.0)
/usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 
47.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 88.3.10)
python (architecture i386):
/Library/Frameworks/Python.framework/Versions/2.7/Python
(compatibility version 2.7.0, current version 2.7.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version 88.3.10)

(compatibility version of libSystem???)

But I imagine there is a cleaner way -- Ronald??

-Chris






-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Binary Wheels and universal builds on OS-X

[Chris Barker - NOAA Federal]

On Tue, Jun 4, 2013 at 10:06 AM, Ronald Oussoren ronaldousso...@mac.com wrote:
 This is not ideal, but works good enough for now. In the long run this 
 should likely be replaced by the compressed tags sets from PEP 425 (at the 
 cost of a much longer file names).

 I think now may be the long run.

 Maybe, at least for 3.4 (assuming that wheel support gets in the stdlib by 
 then).

yup.

 I'd currently prefer to only install binaries that exactly match the python 
 installation, that's the most conservative solution, requires no changes to 
 installation tools and is easier to reason about.

we've got a consensus of two now -- good enough for me ;-)

 All that being said, I sometimes think that we should simply have a 32
 bit and 64 bit binary out there, and forget all this universal stuff!
 It works OK for Windows

 For some definition of OK ;-), the registry can be fun when you're running 32 
 binaries on a 64-bit windows.  A clear advantage of universal binaries is 
 that you can have one set of binaries and don't have to pick which 
 architecture gets the most obvious installation directory (e.g. /lib vs. 
 /lib64 on a lot of Linux systems).

yeah -- I do like them, when they work...

 So we may want to update distutils.util.get_platform to do a PEP 425
 name, but we've got something that could work now.

 That can wait until the new packaging stuff (wheels, metadata 2.0, distlib, 
 ...) gets merged, and even then it might be nicer to keep 'fat' and 'intel' 
 as short names.

it would be best for any change to get merged around the same time as
the packaging stuff -- so there is only one versin out there. But I
agree, the short names are nice -- if the only place anyone would get
the names is distutils, then we're fine.

This is all closer that I thought was -- nice!

-Chris





 Ronald




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Binary Wheels and universal builds on OS-X

[Chris Barker - NOAA Federal]

On Tue, Jun 4, 2013 at 9:55 AM, Ronald Oussoren ronaldousso...@mac.com wrote:

 $ otool -L python
 python (architecture ppc):
   /Library/Frameworks/Python.framework/Versions/2.7/Python
 (compatibility version 2.7.0, current version 2.7.0)
   /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 
 47.1.0)
   /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
 version 88.3.10)
 python (architecture i386):
   /Library/Frameworks/Python.framework/Versions/2.7/Python
 (compatibility version 2.7.0, current version 2.7.0)
   /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
 version 88.3.10)

 (compatibility version of libSystem???)

 But I imagine there is a cleaner way -- Ronald??

 The output of 'otool -l' lists (amongst others) the value of 
 LC_VERSION_MIN_MACOSX which is the deployment target.

I don't see that on my machine -- is that l, a lower-case el'?

 Both can also be found using macholib ;-)

probably best not to add another dependency.

But I've lot track on why we'd need this -- presumably the wheel
builder would know what it built, and could supply the proper
meta-data, and that would be that.

-Chris





-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Binary Wheels and universal builds on OS-X

[Chris Barker - NOAA Federal]

On Fri, May 31, 2013 at 3:38 PM, Daniel Holth dho...@gmail.com wrote:
 Wheel already supports compound tags. Just like py2.py3-none-any a tag

Is this from PEP 425? It's still a little unclear to me exactly how
those tags are to be used, but, yes, this seems to be the right
direction.

 py3-none-x86.ppc (with real platform names) would work. Does that make sense
 for Mac?

hmm, for one, Im confused about the Python Tag vs. the ABI Tag
vs. the Platform Tag -- I guess these all get mushed together in the
filename?

but a few issues anyway: it looks like the platform tag has the OS and
a architecture in one, so would a universal bild on the mac be:

macosx_10_3_i386.ppc
or
macosx_10_3_i386.macosx_10_3_ppc

I'm thinking the latter, though it gets pretty wordy:

A potential full name:

my_package-3_2-cp27-cp27-macosx_10_3_i386.macosx_10_3_ppc.whl

(Im a little confused about python version tag vs. ABI tag, in the
case of a compiled extension...)

But a potential confusion -- in this case, having two platform tags
indicates that _both_ are present, rather than one thing that works
with either. Maybe that would always be the case with platform.

So if that's how the wheel is specified, we still need the other side
of the process:

Installing:

A universal macpython binary needs to know that it is universal
(distutils.util.get_platform() returns only the currently running
version). Once that is know, the installer needs to make some choice:

If a matching universal binary is found -- use it
If a non-universal binary is found that matches the currently running
python -- then what? install it with a warning? raise an exception?

-Chris











 On May 31, 2013 6:30 PM, Chris Barker - NOAA Federal
 chris.bar...@noaa.gov wrote:

 HI Folks,

 A few of us over on the pythonmac list have been hashing out how best
 to support binary packages on OS-X. The binary wheel option seems very
 promising.

 However, there is one Mac-specific issue that does not seem to be
 addressed:

 On OS-X, binaries can be universal -- what this means is that they
 have more than one binary architecture embedded in a single file; the
 system selects the right one at run time. Both executables and dynamic
 libraries can be universal. In theory, an universal binary can
 currently contain up to 4 architectures:

 32+64 bit PPC and 32+64bit x86

 In practice, 64 bit PPC was never broadly used, and 32bit PPC is
 fading fast. Nevertheless, both 32 and 64 Intel systems are pretty
 common, and who knows what there may be in the future (you never know
 with Apple...)

 The binary builds of Python served up on the python.org web site have
 supported universal builds for years -- currently there are two
 options: 32bit PPC+x86 or 32+64bit x86. It's not a single build, as it
 turns out it's hard to support both up to date x86_64 and PPC with the
 same compiler. In these builds, the configuration is set up so that
 distutils will build universal extensions that match the architectures
 included in the builds. This makes it pretty easy to build binary
 packages, and/or full app distributions (py2app) that are universal as
 well.

 setuptools' easy_install supports binary eggs, but always choked on
 universal builds -- it didn't know how to identify what matched the
 system. It would be really nice if future tools could identify and
 install the correct binary wheels for universal builds.

 I've taken a look at PEP 427, and see this:

 platform tag
 E.g. 'linux_x86_64', 'any'.

 That looks to me like there is a built-in assumption that a wheel is
 either specific to a single architecture, or any -- so no way to
 specify that it may have multiple architectures. So I propose that
 platform tag allow a list of some sort:

 ['macosx-10_6_i386', 'macosx_10_6-x86_64']

 or, I suppose to keep it simple, a single string:

 'macosx_10_6_i386+macosx_10_6_x86_64'
 or
 'macosx_10_6_i386+x86_64'

 If so, then a binary wheel could be built (and discovered) that
 supported multiple architectures.

 Then how would an installer ( pip? ) identify the right one? This is a
 bit tricky. PEP 425 states:

 The platform tag is simply distutils.util.get_platform() with all
 hyphens - and periods . replaced with underscore

 On my system right now, I have 32bitPPC + 32bit i386 universal python,
 running on an intel system:

 In [2]: distutils.util.get_platform()
 Out[2]: 'macosx-10.3-i386'

 So we may want to patch get_platform() to support universal builds -
 or add anther function or something -- you may sometimes what to know
 what you are running right now, and other times want to know how the
 current python is built. (note that while distutils can build
 universal binaries, it's not very smart about it -- all is does is
 grab the compiler and linker flags from the Makefile, which include
 multiple -arch flags.

 Even if there is an easy way to query the system, that leaves the
 question of what to do. If a user wants to intall a package into a
 universal python

Re: [Distutils] Binary Wheels and universal builds on OS-X

[Chris Barker - NOAA Federal]

And Ned,

Thanks for your offer to write something up -- looking forward to it.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

Re: [Distutils] Binary Wheels and universal builds on OS-X

[Chris Barker - NOAA Federal]

Darn! this slipped off the list -- I hate it when lists aren't set
with the reply-to address as the list...

On Mon, Jun 3, 2013 at 3:47 PM, Daniel Holth dho...@gmail.com wrote:
 ppc is not completely dead?

I have one under my desk, though truth be told, I haven't turned it on
in a good while...

Which is a good point, of course, we really only have one type of
universal build to support, and even that one  may be obsolete
before too long -- not sure how many 32bit only macs are still around.

 That would make sense. Can you come up with code to detect that a
 newly compiled extension is universal, and that a Python is?

can it be platform-dependent? i.e. use a mac-only system call?

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Fwd: Binary Wheels and universal builds on OS-X

[Chris Barker - NOAA Federal]

Folks,

Accidentally let this slip off the list...

-Chris


-- Forwarded message --
From: Chris Barker - NOAA Federal chris.bar...@noaa.gov
Date: Mon, Jun 3, 2013 at 1:33 PM
Subject: Re: [Distutils] Binary Wheels and universal builds on OS-X
To: Daniel Holth dho...@gmail.com


On Mon, Jun 3, 2013 at 9:57 AM, Daniel Holth dho...@gmail.com wrote:

 A potential full name:

 my_package-3_2-cp27-cp27-macosx_10_3_i386.macosx_10_3_ppc.whl

 (Im a little confused about python version tag vs. ABI tag, in the
 case of a compiled extension...)

 The Python Tag is the Python implementation or language version. It
 lets you indicate compatibility with only CPython or Jython or PyPy
 even if you do not have compiled extensions:

so, in this case, is is right to put cp27 in there twice?

 The platform tag is the OS+architecture.

yeah, kind of wish these were separate, but not really a big deal.

 For compatibility matching the tags are expanded to the Cartesian
 product of the string.split('.') of each of the python, abi, and
 platform tags. This is mostly only useful when only one part of the
 tag is compound.
 http://www.python.org/dev/peps/pep-0425/#compressed-tag-sets

OK -- I think I've got it...

 If this doesn't work then it might be more useful to invent a tag like
 macosx_10_3_universal for some value of universal.

hmm -- the trick is that universal implies that the binary is, well,
universal, but what t really means is has more than one architecture
in it. In practice, there are only  a few combinations likely to see
use, so maybe just defining a few (only two, now, on the python.org
site...) tags makes sense:

macosx_10_3_i386_ppc
and
macosx_10_6_i386_x86_64

-Chris






--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig

[Distutils] Binary Wheels and universal builds on OS-X

[Chris Barker - NOAA Federal]

HI Folks,

A few of us over on the pythonmac list have been hashing out how best
to support binary packages on OS-X. The binary wheel option seems very
promising.

However, there is one Mac-specific issue that does not seem to be addressed:

On OS-X, binaries can be universal -- what this means is that they
have more than one binary architecture embedded in a single file; the
system selects the right one at run time. Both executables and dynamic
libraries can be universal. In theory, an universal binary can
currently contain up to 4 architectures:

32+64 bit PPC and 32+64bit x86

In practice, 64 bit PPC was never broadly used, and 32bit PPC is
fading fast. Nevertheless, both 32 and 64 Intel systems are pretty
common, and who knows what there may be in the future (you never know
with Apple...)

The binary builds of Python served up on the python.org web site have
supported universal builds for years -- currently there are two
options: 32bit PPC+x86 or 32+64bit x86. It's not a single build, as it
turns out it's hard to support both up to date x86_64 and PPC with the
same compiler. In these builds, the configuration is set up so that
distutils will build universal extensions that match the architectures
included in the builds. This makes it pretty easy to build binary
packages, and/or full app distributions (py2app) that are universal as
well.

setuptools' easy_install supports binary eggs, but always choked on
universal builds -- it didn't know how to identify what matched the
system. It would be really nice if future tools could identify and
install the correct binary wheels for universal builds.

I've taken a look at PEP 427, and see this:

platform tag
E.g. 'linux_x86_64', 'any'.

That looks to me like there is a built-in assumption that a wheel is
either specific to a single architecture, or any -- so no way to
specify that it may have multiple architectures. So I propose that
platform tag allow a list of some sort:

['macosx-10_6_i386', 'macosx_10_6-x86_64']

or, I suppose to keep it simple, a single string:

'macosx_10_6_i386+macosx_10_6_x86_64'
or
'macosx_10_6_i386+x86_64'

If so, then a binary wheel could be built (and discovered) that
supported multiple architectures.

Then how would an installer ( pip? ) identify the right one? This is a
bit tricky. PEP 425 states:

The platform tag is simply distutils.util.get_platform() with all
hyphens - and periods . replaced with underscore

On my system right now, I have 32bitPPC + 32bit i386 universal python,
running on an intel system:

In [2]: distutils.util.get_platform()
Out[2]: 'macosx-10.3-i386'

So we may want to patch get_platform() to support universal builds -
or add anther function or something -- you may sometimes what to know
what you are running right now, and other times want to know how the
current python is built. (note that while distutils can build
universal binaries, it's not very smart about it -- all is does is
grab the compiler and linker flags from the Makefile, which include
multiple -arch flags.

Even if there is an easy way to query the system, that leaves the
question of what to do. If a user wants to intall a package into a
universal python, will the installer only accept a binary wheel with
all the supported architectures? or one with only the currently
running one?

Anyway, it would be nice to be able to get this stuff to just work
with universal binaries on the Mac.

Thoughts, ideas?

-Chris



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/ORR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Distutils-SIG maillist  -  Distutils-SIG@python.org
http://mail.python.org/mailman/listinfo/distutils-sig