Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-11-06 Thread Nathaniel Smith 
On Mon, Nov 2, 2015 at 5:57 PM, Nathaniel Smith  wrote:
> On Sun, Nov 1, 2015 at 3:16 PM, Ralf Gommers  wrote:
>> 2. ``pip install .`` silences build output, which may make sense for some
>> usecases, but for numpy it just sits there for minutes with no output after
>> printing "Running setup.py install for numpy". Users will think it hangs and
>> Ctrl-C it. https://github.com/pypa/pip/issues/2732
>
> I tend to agree with the commentary there that for end users this is
> different but no worse than the current situation where we spit out
> pages of "errors" that don't mean anything :-). I posted a suggestion
> on that bug that might help with the apparent hanging problem.

For the record, this is now fixed in pip's "develop" branch and should
be in the next release. For commands like 'setup.py install', pip now
displays a spinner that ticks over whenever the underlying process
prints to stdout/stderr. So if the underlying process hangs, then the
spinner will stop (it's not just lying to you), but normally it works
nicely.

https://github.com/pypa/pip/pull/3224

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-11-02 Thread Chris Barker 
On Tue, Oct 27, 2015 at 8:25 AM, Nathan Goldbaum 
wrote:

> Interestingly, conda actually does "setup.py install" in the recipe for
> numpy:
>

indeed -- many, many conda packages do setup.py install, whihc doesn't mean
it's a good idea --personally, I'm trying hard to switch them all to:

pip install ./

:-)

Which reminds me, the conda skelaton command craes a pip install build.sh
file -- I really need to submit a PR for that ...

There are two cases where a 'pip install' run might go off and start
>> downloading packages without asking you:
>>
>
for my part, regular old setup.py isntall oftem goes off and istalls sutff
too - using easy_install, which really sucks...

This is making me want a setuptools-lite again  -- see the distutils SIG if
you're curious.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-11-02 Thread Nathaniel Smith 
[Adding distutils-sig to the CC as a heads-up. The context is that
numpy is looking at deprecating the use of 'python setup.py install'
and enforcing the use of 'pip install .' instead, and running into
some issues that will probably need to be addressed if 'pip install .'
is going to become the standard interface to work with source trees.]

On Sun, Nov 1, 2015 at 3:16 PM, Ralf Gommers  wrote:
[...]
> Hmm, after some more testing I'm going to have to bring up a few concerns
> myself:
>
> 1. ``pip install .`` still has a clear bug; it starts by copying everything
> (including .git/ !) to a tempdir with shutil, which is very slow. And the
> fix for that will go via ``setup.py sdist``, which is still slow.

Ugh. If 'pip (install/wheel) .' is supposed to become the standard way
to build things, then it should probably build in-place by default.
Working in a temp dir makes perfect sense for 'pip install
' or 'pip install ', but if the user supplies an
actual named on-disk directory then presumably the user is expecting
this directory to be used, and to be able to take advantage of
incremental rebuilds etc., no?

> 2. ``pip install .`` silences build output, which may make sense for some
> usecases, but for numpy it just sits there for minutes with no output after
> printing "Running setup.py install for numpy". Users will think it hangs and
> Ctrl-C it. https://github.com/pypa/pip/issues/2732

I tend to agree with the commentary there that for end users this is
different but no worse than the current situation where we spit out
pages of "errors" that don't mean anything :-). I posted a suggestion
on that bug that might help with the apparent hanging problem.

> 3. ``pip install .`` refuses to upgrade an already installed development
> version. For released versions that makes sense, but if I'm in a git tree
> then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal
> to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).

Ugh, this is clearly just a bug -- `pip install .` should always
unconditionally install, IMO. (Did you file a bug yet?) At least the
workaround is just 'pip uninstall numpy; pip install .', which is
still better the running 'setup.py install' and having it blithely
overwrite some files and not others.

The first and last issue seem like ones that will mostly only affect
developers, who should mostly have the ability to deal with these
weird issues (or just use setup.py install --force if that's what they
prefer)? This still seems like a reasonable trade-off to me if it also
has the effect of reducing the number of weird broken installs among
our thousands-of-times-larger userbase.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-11-01 Thread Charles R Harris 
On Sun, Nov 1, 2015 at 4:16 PM, Ralf Gommers  wrote:

>
>
> On Sun, Nov 1, 2015 at 1:59 AM, Ralf Gommers 
> wrote:
>
>>
>>
>> On Sun, Nov 1, 2015 at 1:54 AM, Ralf Gommers 
>> wrote:
>>
>>>
>>>
>>>
>>> On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser <
>>> warren.weckes...@gmail.com> wrote:
>>>


 On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith 
 wrote:

> Hi all,
>
> Apparently it is not well known that if you have a Python project
> source tree (e.g., a numpy checkout), then the correct way to install
> it is NOT to type
>
>   python setup.py install   # bad and broken!
>
> but rather to type
>
>   pip install .
>
>

 FWIW, I don't see any mention of this in the numpy docs, but I do see a
 lot of instructions involving `setup.py build` and `setup.py install`.
 See, for example, INSTALL.txt.  Also see

 http://docs.scipy.org/doc/numpy/user/install.html#building-from-source
 So I guess it is not surprising that it is not well known.

>>>
>>> Indeed, install docs are always hopelessly outdated. And we have too
>>> many of them. There's duplicate info in INSTALL.txt and
>>> http://scipy.org/scipylib/building/index.html for example. We should
>>> probably just empty out INSTALL.txt and simply put a link in it to the html
>>> docs.
>>>
>>> I've created an issue with a long todo list and a bunch of links:
>>> https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or
>>> to go fix something:)
>>>
>>
>> Oh, and: looking at this thread there haven't been serious unanswered
>> concerns (at least in my perception), so without more discussion I'd
>> interpret the current status as "go ahead".
>>
>
> Hmm, after some more testing I'm going to have to bring up a few concerns
> myself:
>
> 1. ``pip install .`` still has a clear bug; it starts by copying
> everything (including .git/ !) to a tempdir with shutil, which is very
> slow. And the fix for that will go via ``setup.py sdist``, which is still
> slow.
>
> 2. ``pip install .`` silences build output, which may make sense for some
> usecases, but for numpy it just sits there for minutes with no output after
> printing "Running setup.py install for numpy". Users will think it hangs
> and Ctrl-C it. https://github.com/pypa/pip/issues/2732
>
> 3. ``pip install .`` refuses to upgrade an already installed development
> version. For released versions that makes sense, but if I'm in a git tree
> then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal
> to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).
>
>
> I've sent a (incomplete) fix for the shutil thing (
> https://github.com/pypa/pip/pull/3219) and will comment on some open
> issues on the pip tracker. But I'm thinking that for now we should go with
> some printed message first. Something like "please use ``pip install .`` if
> you want reliable uninstall behavior. See  for more details".
>
> Pip has worked quite well for me in the past, but the above makes me thing
> it's not much of an improvement over use of setuptools.
>

Which version of pip?

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-11-01 Thread Ralf Gommers 
On Sun, Nov 1, 2015 at 1:59 AM, Ralf Gommers  wrote:

>
>
> On Sun, Nov 1, 2015 at 1:54 AM, Ralf Gommers 
> wrote:
>
>>
>>
>>
>> On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser <
>> warren.weckes...@gmail.com> wrote:
>>
>>>
>>>
>>> On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith  wrote:
>>>
 Hi all,

 Apparently it is not well known that if you have a Python project
 source tree (e.g., a numpy checkout), then the correct way to install
 it is NOT to type

   python setup.py install   # bad and broken!

 but rather to type

   pip install .


>>>
>>> FWIW, I don't see any mention of this in the numpy docs, but I do see a
>>> lot of instructions involving `setup.py build` and `setup.py install`.
>>> See, for example, INSTALL.txt.  Also see
>>>
>>> http://docs.scipy.org/doc/numpy/user/install.html#building-from-source
>>> So I guess it is not surprising that it is not well known.
>>>
>>
>> Indeed, install docs are always hopelessly outdated. And we have too many
>> of them. There's duplicate info in INSTALL.txt and
>> http://scipy.org/scipylib/building/index.html for example. We should
>> probably just empty out INSTALL.txt and simply put a link in it to the html
>> docs.
>>
>> I've created an issue with a long todo list and a bunch of links:
>> https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or
>> to go fix something:)
>>
>
> Oh, and: looking at this thread there haven't been serious unanswered
> concerns (at least in my perception), so without more discussion I'd
> interpret the current status as "go ahead".
>

Hmm, after some more testing I'm going to have to bring up a few concerns
myself:

1. ``pip install .`` still has a clear bug; it starts by copying everything
(including .git/ !) to a tempdir with shutil, which is very slow. And the
fix for that will go via ``setup.py sdist``, which is still slow.

2. ``pip install .`` silences build output, which may make sense for some
usecases, but for numpy it just sits there for minutes with no output after
printing "Running setup.py install for numpy". Users will think it hangs
and Ctrl-C it. https://github.com/pypa/pip/issues/2732

3. ``pip install .`` refuses to upgrade an already installed development
version. For released versions that makes sense, but if I'm in a git tree
then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal
to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).


I've sent a (incomplete) fix for the shutil thing (
https://github.com/pypa/pip/pull/3219) and will comment on some open issues
on the pip tracker. But I'm thinking that for now we should go with some
printed message first. Something like "please use ``pip install .`` if you
want reliable uninstall behavior. See  for more details".

Pip has worked quite well for me in the past, but the above makes me thing
it's not much of an improvement over use of setuptools.

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-11-01 Thread Ralf Gommers 
On Mon, Nov 2, 2015 at 1:12 AM, Charles R Harris 
wrote:

>
>
> On Sun, Nov 1, 2015 at 4:16 PM, Ralf Gommers 
> wrote:
>
>>
>>
>> On Sun, Nov 1, 2015 at 1:59 AM, Ralf Gommers 
>> wrote:
>>
>>>
>>>
>>> On Sun, Nov 1, 2015 at 1:54 AM, Ralf Gommers 
>>> wrote:
>>>



 On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser <
 warren.weckes...@gmail.com> wrote:

>
>
> On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith 
> wrote:
>
>> Hi all,
>>
>> Apparently it is not well known that if you have a Python project
>> source tree (e.g., a numpy checkout), then the correct way to install
>> it is NOT to type
>>
>>   python setup.py install   # bad and broken!
>>
>> but rather to type
>>
>>   pip install .
>>
>>
>
> FWIW, I don't see any mention of this in the numpy docs, but I do see
> a lot of instructions involving `setup.py build` and `setup.py install`.
> See, for example, INSTALL.txt.  Also see
>
> http://docs.scipy.org/doc/numpy/user/install.html#building-from-source
> So I guess it is not surprising that it is not well known.
>

 Indeed, install docs are always hopelessly outdated. And we have too
 many of them. There's duplicate info in INSTALL.txt and
 http://scipy.org/scipylib/building/index.html for example. We should
 probably just empty out INSTALL.txt and simply put a link in it to the html
 docs.

 I've created an issue with a long todo list and a bunch of links:
 https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or
 to go fix something:)

>>>
>>> Oh, and: looking at this thread there haven't been serious unanswered
>>> concerns (at least in my perception), so without more discussion I'd
>>> interpret the current status as "go ahead".
>>>
>>
>> Hmm, after some more testing I'm going to have to bring up a few concerns
>> myself:
>>
>> 1. ``pip install .`` still has a clear bug; it starts by copying
>> everything (including .git/ !) to a tempdir with shutil, which is very
>> slow. And the fix for that will go via ``setup.py sdist``, which is still
>> slow.
>>
>> 2. ``pip install .`` silences build output, which may make sense for some
>> usecases, but for numpy it just sits there for minutes with no output after
>> printing "Running setup.py install for numpy". Users will think it hangs
>> and Ctrl-C it. https://github.com/pypa/pip/issues/2732
>>
>> 3. ``pip install .`` refuses to upgrade an already installed development
>> version. For released versions that makes sense, but if I'm in a git tree
>> then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal
>> to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).
>>
>>
>> I've sent a (incomplete) fix for the shutil thing (
>> https://github.com/pypa/pip/pull/3219) and will comment on some open
>> issues on the pip tracker. But I'm thinking that for now we should go with
>> some printed message first. Something like "please use ``pip install .`` if
>> you want reliable uninstall behavior. See  for more details".
>>
>> Pip has worked quite well for me in the past, but the above makes me
>> thing it's not much of an improvement over use of setuptools.
>>
>
> Which version of pip?
>

Latest master (it's 'develop' branch). Recent released versions will be the
same, because there are open issues for these things.

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-31 Thread Ralf Gommers 
On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser <
warren.weckes...@gmail.com> wrote:

>
>
> On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith  wrote:
>
>> Hi all,
>>
>> Apparently it is not well known that if you have a Python project
>> source tree (e.g., a numpy checkout), then the correct way to install
>> it is NOT to type
>>
>>   python setup.py install   # bad and broken!
>>
>> but rather to type
>>
>>   pip install .
>>
>>
>
> FWIW, I don't see any mention of this in the numpy docs, but I do see a
> lot of instructions involving `setup.py build` and `setup.py install`.
> See, for example, INSTALL.txt.  Also see
> http://docs.scipy.org/doc/numpy/user/install.html#building-from-source
> So I guess it is not surprising that it is not well known.
>

Indeed, install docs are always hopelessly outdated. And we have too many
of them. There's duplicate info in INSTALL.txt and
http://scipy.org/scipylib/building/index.html for example. We should
probably just empty out INSTALL.txt and simply put a link in it to the html
docs.

I've created an issue with a long todo list and a bunch of links:
https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or to
go fix something:)

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-31 Thread Ralf Gommers 
On Sun, Nov 1, 2015 at 1:54 AM, Ralf Gommers  wrote:

>
>
>
> On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser <
> warren.weckes...@gmail.com> wrote:
>
>>
>>
>> On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith  wrote:
>>
>>> Hi all,
>>>
>>> Apparently it is not well known that if you have a Python project
>>> source tree (e.g., a numpy checkout), then the correct way to install
>>> it is NOT to type
>>>
>>>   python setup.py install   # bad and broken!
>>>
>>> but rather to type
>>>
>>>   pip install .
>>>
>>>
>>
>> FWIW, I don't see any mention of this in the numpy docs, but I do see a
>> lot of instructions involving `setup.py build` and `setup.py install`.
>> See, for example, INSTALL.txt.  Also see
>>
>> http://docs.scipy.org/doc/numpy/user/install.html#building-from-source
>> So I guess it is not surprising that it is not well known.
>>
>
> Indeed, install docs are always hopelessly outdated. And we have too many
> of them. There's duplicate info in INSTALL.txt and
> http://scipy.org/scipylib/building/index.html for example. We should
> probably just empty out INSTALL.txt and simply put a link in it to the html
> docs.
>
> I've created an issue with a long todo list and a bunch of links:
> https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or to
> go fix something:)
>

Oh, and: looking at this thread there haven't been serious unanswered
concerns (at least in my perception), so without more discussion I'd
interpret the current status as "go ahead".

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-29 Thread Todd 
On Oct 29, 2015 00:29, "Sandro Tosi"  wrote:
>
> please, pretty please, do not disable setup.py install or at least
> keep providing a way for distribution (Debian in this case) to be able
> to build/install numpy in a temporary location for packaging reasons.
> pip is not the solution for us

What is wrong with "pip install --root" ?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-28 Thread Jerome Kieffer 
On Tue, 27 Oct 2015 15:35:50 -0700 (PDT)
"Juan Nunez-Iglesias"  wrote:

> Can someone here who understands more about distribution maybe write a blog 
> post detailing:

Hi,

Olivier Grisel from sklearn gave a very good talk on this topic at PyCon, 
earlier
this year:
http://www.pyvideo.org/video/3473/build-and-test-wheel-packages-on-linux-osx-win

Very instructive.
-- 
Jérôme Kieffer
tel +33 476 882 445
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-28 Thread Sandro Tosi 
please, pretty please, do not disable setup.py install or at least
keep providing a way for distribution (Debian in this case) to be able
to build/install numpy in a temporary location for packaging reasons.
pip is not the solution for us

On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith  wrote:
> Hi all,
>
> Apparently it is not well known that if you have a Python project
> source tree (e.g., a numpy checkout), then the correct way to install
> it is NOT to type
>
>   python setup.py install   # bad and broken!
>
> but rather to type
>
>   pip install .
>
> (I.e., pip install isn't just for packages on pypi -- you can also
> pass it the path to an arbitrary source directory or the URL of a
> source tarball and it will do its thing. In this case "install ."
> means "install the project in the current directory".)
>
> These don't quite have identical results -- the main difference is
> that the latter makes sure that proper metadata gets installed so that
> later on it will be possible to upgrade or uninstall correctly. If you
> call setup.py directly, and then later you try to upgrade your
> package, then it's entirely possible to end up with a mixture of old
> and new versions of the package installed in your PYTHONPATH. (One
> common effect is in numpy's case is that we get people sending us
> mysterious bug reports about failing tests in files don't even exist
> (!) -- because nose is finding tests in files from one version of
> numpy and running them against a different version of numpy.)
>
> But this isn't the only issue -- using pip also avoids a bunch of
> weird corner cases in distutils/setuptools. E.g., if setup.py uses
> plain distutils, then it turns out this will mangle numpy version
> numbers in ways that cause weird horribleness -- see [1] for a bug
> report of the form "matplotlib doesn't build anymore" which turned out
> to be because of using 'setup.py install' to install numpy. OTOH if
> setup.py uses setuptools then you get different weirdnesses, like you
> can easily end up with multiple versions of the same library installed
> simultaneously.
>
> And finally, an advantage of getting used to using 'pip install .' now
> is that you'll be prepared for the glorious future when we kill
> distutils and get rid of setup.py entirely in favor of something less
> terrible [2].
>
> So a proposal that came out of the discussion in [1] is that we modify
> numpy's setup.py now so that if you try running
>
> python setup.py install
>
> you get
>
> Error: Calling 'setup.py install' directly is NOT SUPPORTED!
> Instead, do:
>
> pip install .
>
> Alternatively, if you want to proceed at your own risk, you
> can try 'setup.py install --force-raw-setup.py'
> For more information see http://...
>
> (Other setup.py commands would continue to work as normal.)
>
> I believe that this would also break both 'easy_install numpy', and
> attempts to install numpy via the setup_requires= argument to
> setuptools.setup (because setup_requires= implicitly calls
> easy_install). install_requires= would *not* be affected, and
> setup_requires= would still be fine in cases where numpy was already
> installed.
>
> This would hopefully cut down on the amount of time everyone spends
> trying to track down these stupid weird bugs, but it will also require
> some adjustment in people's workflows, so... objections? concerns?
>
> -n
>
> [1] https://github.com/numpy/numpy/issues/6551
> [2] https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion



-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-28 Thread Ralf Gommers 
On Thu, Oct 29, 2015 at 12:28 AM, Sandro Tosi  wrote:

> please, pretty please, do not disable setup.py install or at least
> keep providing a way for distribution (Debian in this case) to be able
> to build/install numpy in a temporary location for packaging reasons.
> pip is not the solution for us
>

``python setup.py install --force`` would still work. Would that be OK then?

Ralf


> On Tue, Oct 27, 2015 at 12:31 AM, Nathaniel Smith  wrote:
> > Hi all,
> >
> > Apparently it is not well known that if you have a Python project
> > source tree (e.g., a numpy checkout), then the correct way to install
> > it is NOT to type
> >
> >   python setup.py install   # bad and broken!
> >
> > but rather to type
> >
> >   pip install .
> >
> > (I.e., pip install isn't just for packages on pypi -- you can also
> > pass it the path to an arbitrary source directory or the URL of a
> > source tarball and it will do its thing. In this case "install ."
> > means "install the project in the current directory".)
> >
> > These don't quite have identical results -- the main difference is
> > that the latter makes sure that proper metadata gets installed so that
> > later on it will be possible to upgrade or uninstall correctly. If you
> > call setup.py directly, and then later you try to upgrade your
> > package, then it's entirely possible to end up with a mixture of old
> > and new versions of the package installed in your PYTHONPATH. (One
> > common effect is in numpy's case is that we get people sending us
> > mysterious bug reports about failing tests in files don't even exist
> > (!) -- because nose is finding tests in files from one version of
> > numpy and running them against a different version of numpy.)
> >
> > But this isn't the only issue -- using pip also avoids a bunch of
> > weird corner cases in distutils/setuptools. E.g., if setup.py uses
> > plain distutils, then it turns out this will mangle numpy version
> > numbers in ways that cause weird horribleness -- see [1] for a bug
> > report of the form "matplotlib doesn't build anymore" which turned out
> > to be because of using 'setup.py install' to install numpy. OTOH if
> > setup.py uses setuptools then you get different weirdnesses, like you
> > can easily end up with multiple versions of the same library installed
> > simultaneously.
> >
> > And finally, an advantage of getting used to using 'pip install .' now
> > is that you'll be prepared for the glorious future when we kill
> > distutils and get rid of setup.py entirely in favor of something less
> > terrible [2].
> >
> > So a proposal that came out of the discussion in [1] is that we modify
> > numpy's setup.py now so that if you try running
> >
> > python setup.py install
> >
> > you get
> >
> > Error: Calling 'setup.py install' directly is NOT SUPPORTED!
> > Instead, do:
> >
> > pip install .
> >
> > Alternatively, if you want to proceed at your own risk, you
> > can try 'setup.py install --force-raw-setup.py'
> > For more information see http://...
> >
> > (Other setup.py commands would continue to work as normal.)
> >
> > I believe that this would also break both 'easy_install numpy', and
> > attempts to install numpy via the setup_requires= argument to
> > setuptools.setup (because setup_requires= implicitly calls
> > easy_install). install_requires= would *not* be affected, and
> > setup_requires= would still be fine in cases where numpy was already
> > installed.
> >
> > This would hopefully cut down on the amount of time everyone spends
> > trying to track down these stupid weird bugs, but it will also require
> > some adjustment in people's workflows, so... objections? concerns?
> >
> > -n
> >
> > [1] https://github.com/numpy/numpy/issues/6551
> > [2]
> https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html
> >
> > --
> > Nathaniel J. Smith -- http://vorpus.org
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> --
> Sandro Tosi (aka morph, morpheus, matrixhasu)
> My website: http://matrixhasu.altervista.org/
> Me at Debian: http://wiki.debian.org/SandroTosi
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-28 Thread Juan Nunez-Iglesias 
Thanks, Jerome! I’ve added it to my to-watch list. It sounds really useful!




Juan.

On Wed, Oct 28, 2015 at 6:36 PM, Jerome Kieffer 
wrote:

> On Tue, 27 Oct 2015 15:35:50 -0700 (PDT)
> "Juan Nunez-Iglesias"  wrote:
>> Can someone here who understands more about distribution maybe write a blog 
>> post detailing:
> Hi,
> Olivier Grisel from sklearn gave a very good talk on this topic at PyCon, 
> earlier
> this year:
> http://www.pyvideo.org/video/3473/build-and-test-wheel-packages-on-linux-osx-win
> Very instructive.
> -- 
> Jérôme Kieffer
> tel +33 476 882 445
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Nathaniel Smith 
On Mon, Oct 26, 2015 at 11:53 PM, Juan Nunez-Iglesias
 wrote:
> Is there a pip equivalent of "python setup.py develop"?

Kinda answered this already when replying to Chuck, but: yes, it's
`pip install -e ` (the -e is short for
--editable), not that you would need it necessarily because `setup.py
develop` would still be legal.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Ralf Gommers 
On Tue, Oct 27, 2015 at 6:44 AM, Nathaniel Smith  wrote:

> On Mon, Oct 26, 2015 at 9:31 PM, Nathaniel Smith  wrote:
> [...]
> > I believe that this would also break both 'easy_install numpy', and
> > attempts to install numpy via the setup_requires= argument to
> > setuptools.setup (because setup_requires= implicitly calls
> > easy_install). install_requires= would *not* be affected, and
> > setup_requires= would still be fine in cases where numpy was already
> > installed.
>
> On further investigation, it looks like the simplest approach to doing
> this would actually treat easy_install and setup_requires= the same
> way as they treat pip, i.e., they would all be allowed. (I was
> misreading some particularly confusing code in setuptools.)
>
> It also looks like easy_installed packages can at least be safely
> upgraded, so I guess allowing this is okay :-).


I just discovered https://bitbucket.org/dholth/setup-requires, which
ensures that setup_requires uses pip instead of easy_install. So we can not
only keep setup-requires working, but make it work significantly better.

So if/when we accept the proposal in this thread, I'm thinking we should
make a bunch of changes at once:
- always use setuptools (this is a new dependency)
- error on ``python setup.py install``
- add the setup-requires trick
- error on ``python setup.py clean`` (saying "use `git clean -xdf` (or -Xdf
...) instead")
- change ``python setup.py --help`` to first show numpy-specific stuff
before setuptools help info
- update all our install docs

And when "pip upgrade" is released (should be soon, see
https://github.com/pypa/pip/pull/3194), officially change our mind and
recommend the use of install_requires/setup_requires to packages depending
on numpy.

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Charles R Harris 
On Tue, Oct 27, 2015 at 12:08 AM, Nathaniel Smith  wrote:

> On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris
>  wrote:
> >
> [...]
> > I gave it a shot the other day. Pip keeps a record of the path to the
> repo
> > and in order to cleanup I needed to search out the file and delete the
> repo
> > path. There is probably a better way to do that, but it didn't strike me
> as
> > less troublesome than ` python setup.py install --local`.
>
> Sorry, what did you "give a shot", and what problem did it create?
> What does `setup.py install --local` do? (it doesn't seem to be
> mentioned in `setup.py install --help`.)
>

 `pip install --user -e . `. However, `pip install --user .` seems to work
fine. The pip documentation isn't the best.

Yeah, `--user` not `--local`. It's getting late...

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Ralf Gommers 
On Tue, Oct 27, 2015 at 8:28 AM, Nathaniel Smith  wrote:

> On Tue, Oct 27, 2015 at 12:19 AM, Ralf Gommers 
> wrote:
> >
> >
> > On Tue, Oct 27, 2015 at 6:44 AM, Nathaniel Smith  wrote:
> >>
> >> On Mon, Oct 26, 2015 at 9:31 PM, Nathaniel Smith  wrote:
> >> [...]
> >> > I believe that this would also break both 'easy_install numpy', and
> >> > attempts to install numpy via the setup_requires= argument to
> >> > setuptools.setup (because setup_requires= implicitly calls
> >> > easy_install). install_requires= would *not* be affected, and
> >> > setup_requires= would still be fine in cases where numpy was already
> >> > installed.
> >>
> >> On further investigation, it looks like the simplest approach to doing
> >> this would actually treat easy_install and setup_requires= the same
> >> way as they treat pip, i.e., they would all be allowed. (I was
> >> misreading some particularly confusing code in setuptools.)
> >>
> >> It also looks like easy_installed packages can at least be safely
> >> upgraded, so I guess allowing this is okay :-).
> >
> >
> > I just discovered https://bitbucket.org/dholth/setup-requires, which
> ensures
> > that setup_requires uses pip instead of easy_install. So we can not only
> > keep setup-requires working, but make it work significantly better.
>
> IIUC this is not something that we (= numpy) could use ourselves, but
> instead something that everyone who does setup_requires=["numpy"]
> would have to set up in their individual projects?
>

Right. I was thinking about using it in scipy. Ah well, I'm sure we can
manage to not break ``setup_requires=numpy`` in some way.

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Juan Nunez-Iglesias 
Is there a pip equivalent of "python setup.py develop"?

On Tue, Oct 27, 2015 at 5:33 PM Charles R Harris 
wrote:

> On Tue, Oct 27, 2015 at 12:08 AM, Nathaniel Smith  wrote:
>
>> On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris
>>  wrote:
>> >
>> [...]
>> > I gave it a shot the other day. Pip keeps a record of the path to the
>> repo
>> > and in order to cleanup I needed to search out the file and delete the
>> repo
>> > path. There is probably a better way to do that, but it didn't strike
>> me as
>> > less troublesome than ` python setup.py install --local`.
>>
>> Sorry, what did you "give a shot", and what problem did it create?
>> What does `setup.py install --local` do? (it doesn't seem to be
>> mentioned in `setup.py install --help`.)
>>
>
>  `pip install --user -e . `. However, `pip install --user .` seems to work
> fine. The pip documentation isn't the best.
>
> Yeah, `--user` not `--local`. It's getting late...
>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Nathaniel Smith 
On Mon, Oct 26, 2015 at 11:33 PM, Charles R Harris
 wrote:
>
>
> On Tue, Oct 27, 2015 at 12:08 AM, Nathaniel Smith  wrote:
>>
>> On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris
>>  wrote:
>> >
>> [...]
>> > I gave it a shot the other day. Pip keeps a record of the path to the
>> > repo
>> > and in order to cleanup I needed to search out the file and delete the
>> > repo
>> > path. There is probably a better way to do that, but it didn't strike me
>> > as
>> > less troublesome than ` python setup.py install --local`.
>>
>> Sorry, what did you "give a shot", and what problem did it create?
>> What does `setup.py install --local` do? (it doesn't seem to be
>> mentioned in `setup.py install --help`.)
>
>
>  `pip install --user -e . `. However, `pip install --user .` seems to work
> fine. The pip documentation isn't the best.
>
> Yeah, `--user` not `--local`. It's getting late...

Ah. I think if you want to undo a `pip install --user -e .` you can
just do `pip uninstall numpy`? But in any case the equivalent of 'pip
install -e' is 'setup.py develop', and the proposal is only to disable
`setup.py install`. So if you prefer `setup.py develop` then you could
still use it.

(IIUC `pip install -e` is a *very* thin wrapper around `setup.py
develop` -- normally pip takes over the job of actually installing
files, so pip gets to interpret options like --user and figure out
what it means for where it puts the files, but in the case of `install
-e` it looks like it just calls `setup.py develop` directly, so if
--user doesn't work it's probably because setup.py develop doesn't
know about --user?)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Nathaniel Smith 
On Tue, Oct 27, 2015 at 12:19 AM, Ralf Gommers  wrote:
>
>
> On Tue, Oct 27, 2015 at 6:44 AM, Nathaniel Smith  wrote:
>>
>> On Mon, Oct 26, 2015 at 9:31 PM, Nathaniel Smith  wrote:
>> [...]
>> > I believe that this would also break both 'easy_install numpy', and
>> > attempts to install numpy via the setup_requires= argument to
>> > setuptools.setup (because setup_requires= implicitly calls
>> > easy_install). install_requires= would *not* be affected, and
>> > setup_requires= would still be fine in cases where numpy was already
>> > installed.
>>
>> On further investigation, it looks like the simplest approach to doing
>> this would actually treat easy_install and setup_requires= the same
>> way as they treat pip, i.e., they would all be allowed. (I was
>> misreading some particularly confusing code in setuptools.)
>>
>> It also looks like easy_installed packages can at least be safely
>> upgraded, so I guess allowing this is okay :-).
>
>
> I just discovered https://bitbucket.org/dholth/setup-requires, which ensures
> that setup_requires uses pip instead of easy_install. So we can not only
> keep setup-requires working, but make it work significantly better.

IIUC this is not something that we (= numpy) could use ourselves, but
instead something that everyone who does setup_requires=["numpy"]
would have to set up in their individual projects?

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Charles R Harris 
On Mon, Oct 26, 2015 at 10:31 PM, Nathaniel Smith  wrote:

> Hi all,
>
> Apparently it is not well known that if you have a Python project
> source tree (e.g., a numpy checkout), then the correct way to install
> it is NOT to type
>
>   python setup.py install   # bad and broken!
>
> but rather to type
>
>   pip install .
>
> (I.e., pip install isn't just for packages on pypi -- you can also
> pass it the path to an arbitrary source directory or the URL of a
> source tarball and it will do its thing. In this case "install ."
> means "install the project in the current directory".)
>
> These don't quite have identical results -- the main difference is
> that the latter makes sure that proper metadata gets installed so that
> later on it will be possible to upgrade or uninstall correctly. If you
> call setup.py directly, and then later you try to upgrade your
> package, then it's entirely possible to end up with a mixture of old
> and new versions of the package installed in your PYTHONPATH. (One
> common effect is in numpy's case is that we get people sending us
> mysterious bug reports about failing tests in files don't even exist
> (!) -- because nose is finding tests in files from one version of
> numpy and running them against a different version of numpy.)
>
> But this isn't the only issue -- using pip also avoids a bunch of
> weird corner cases in distutils/setuptools. E.g., if setup.py uses
> plain distutils, then it turns out this will mangle numpy version
> numbers in ways that cause weird horribleness -- see [1] for a bug
> report of the form "matplotlib doesn't build anymore" which turned out
> to be because of using 'setup.py install' to install numpy. OTOH if
> setup.py uses setuptools then you get different weirdnesses, like you
> can easily end up with multiple versions of the same library installed
> simultaneously.
>
> And finally, an advantage of getting used to using 'pip install .' now
> is that you'll be prepared for the glorious future when we kill
> distutils and get rid of setup.py entirely in favor of something less
> terrible [2].
>
> So a proposal that came out of the discussion in [1] is that we modify
> numpy's setup.py now so that if you try running
>
> python setup.py install
>
> you get
>
> Error: Calling 'setup.py install' directly is NOT SUPPORTED!
> Instead, do:
>
> pip install .
>
> Alternatively, if you want to proceed at your own risk, you
> can try 'setup.py install --force-raw-setup.py'
> For more information see http://...
>
> (Other setup.py commands would continue to work as normal.)
>
> I believe that this would also break both 'easy_install numpy', and
> attempts to install numpy via the setup_requires= argument to
> setuptools.setup (because setup_requires= implicitly calls
> easy_install). install_requires= would *not* be affected, and
> setup_requires= would still be fine in cases where numpy was already
> installed.
>
> This would hopefully cut down on the amount of time everyone spends
> trying to track down these stupid weird bugs, but it will also require
> some adjustment in people's workflows, so... objections? concerns?
>

I gave it a shot the other day. Pip keeps a record of the path to the repo
and in order to cleanup I needed to search out the file and delete the repo
path. There is probably a better way to do that, but it didn't strike me as
less troublesome than ` python setup.py install --local`.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Nathaniel Smith 
On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris
 wrote:
>
[...]
> I gave it a shot the other day. Pip keeps a record of the path to the repo
> and in order to cleanup I needed to search out the file and delete the repo
> path. There is probably a better way to do that, but it didn't strike me as
> less troublesome than ` python setup.py install --local`.

Sorry, what did you "give a shot", and what problem did it create?
What does `setup.py install --local` do? (it doesn't seem to be
mentioned in `setup.py install --help`.)

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread josef.pktd 
On Tue, Oct 27, 2015 at 3:32 AM, Ralf Gommers 
wrote:

>
>
> On Tue, Oct 27, 2015 at 8:28 AM, Nathaniel Smith  wrote:
>
>> On Tue, Oct 27, 2015 at 12:19 AM, Ralf Gommers 
>> wrote:
>> >
>> >
>> > On Tue, Oct 27, 2015 at 6:44 AM, Nathaniel Smith  wrote:
>> >>
>> >> On Mon, Oct 26, 2015 at 9:31 PM, Nathaniel Smith 
>> wrote:
>> >> [...]
>> >> > I believe that this would also break both 'easy_install numpy', and
>> >> > attempts to install numpy via the setup_requires= argument to
>> >> > setuptools.setup (because setup_requires= implicitly calls
>> >> > easy_install). install_requires= would *not* be affected, and
>> >> > setup_requires= would still be fine in cases where numpy was already
>> >> > installed.
>> >>
>> >> On further investigation, it looks like the simplest approach to doing
>> >> this would actually treat easy_install and setup_requires= the same
>> >> way as they treat pip, i.e., they would all be allowed. (I was
>> >> misreading some particularly confusing code in setuptools.)
>> >>
>> >> It also looks like easy_installed packages can at least be safely
>> >> upgraded, so I guess allowing this is okay :-).
>> >
>> >
>> > I just discovered https://bitbucket.org/dholth/setup-requires, which
>> ensures
>> > that setup_requires uses pip instead of easy_install. So we can not only
>> > keep setup-requires working, but make it work significantly better.
>>
>> IIUC this is not something that we (= numpy) could use ourselves, but
>> instead something that everyone who does setup_requires=["numpy"]
>> would have to set up in their individual projects?
>>
>
> Right. I was thinking about using it in scipy. Ah well, I'm sure we can
> manage to not break ``setup_requires=numpy`` in some way.
>


What's the equivalent of
python setup.py build_ext --inplace


brief google search (I didn't follow up on those)

https://github.com/pypa/pip/issues/1887
https://github.com/pypa/pip/issues/18


Given that I rely completely on binary distributions for numpy and scipy, I
won't be affected.

(I'm still allergic to pip and will switch only several years after
everybody else.)

Josef



>
> Ralf
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread josef.pktd 
On Tue, Oct 27, 2015 at 10:59 AM, Nathaniel Smith  wrote:

> On Oct 27, 2015 6:08 AM,  wrote:
> >
> [...]
> >
> >
> > What's the equivalent of
> > python setup.py build_ext --inplace
>
> It's
> python setup.py build_ext --inplace
>
> ;-)
>
Ok, Sorry, I read now the small print and the issue.

Sounds reasonable, given we can `force` our way out.

(If the reason to run to pip is a misspelled dev version number, then it
looks like a hammer to me.)

Josef



> There's also no replacement for setup.py sdist, or setup.py upload (which
> is broken and should never be used), or setup.py clean (which is also
> broken and should never be used in numpy's case). pip is a better package
> installer than raw distutils or setuptools, for non-installation-related
> tasks it has nothing to offer. (With the partial exception of 'pip wheel'.)
>
> -n
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Nathan Goldbaum 
Would this happen at the level of numpy's setup.py script or would it be
implemented in numpy.distutils? I'm asking as the developer of a package
that uses numpy.distutils to manage C extensions.

On Tue, Oct 27, 2015 at 10:28 AM,  wrote:

>
>
> On Tue, Oct 27, 2015 at 10:59 AM, Nathaniel Smith  wrote:
>
>> On Oct 27, 2015 6:08 AM,  wrote:
>> >
>> [...]
>> >
>> >
>> > What's the equivalent of
>> > python setup.py build_ext --inplace
>>
>> It's
>> python setup.py build_ext --inplace
>>
>> ;-)
>>
> Ok, Sorry, I read now the small print and the issue.
>
> Sounds reasonable, given we can `force` our way out.
>
> (If the reason to run to pip is a misspelled dev version number, then it
> looks like a hammer to me.)
>
> Josef
>
>
>
>> There's also no replacement for setup.py sdist, or setup.py upload (which
>> is broken and should never be used), or setup.py clean (which is also
>> broken and should never be used in numpy's case). pip is a better package
>> installer than raw distutils or setuptools, for non-installation-related
>> tasks it has nothing to offer. (With the partial exception of 'pip wheel'.)
>>
>> -n
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Nathaniel Smith 
On Oct 27, 2015 6:48 AM, "James E.H. Turner"  wrote:
>>
>> Apparently it is not well known that if you have a Python project
>> source tree (e.g., a numpy checkout), then the correct way to install
>> it is NOT to type
>>
>>python setup.py install   # bad and broken!
>>
>> but rather to type
>>
>>pip install .
>
>
> Though I haven't studied it exhaustively, it always seems to me that
> pip is bad & broken, whereas python setup.py install does what I
> expect (even if it's a mess internally).

Unfortunately this is only true if what you expect is for packages to be
installed in subtly corrupted ways, as described in the original email.
Sorry to be the bearer of bad tidings :-/

> In particular, when
> maintaining a distribution of Python packages, you try to have some
> well-defined, reproducible build from source tarballs and then you
> find that pip is going off and downloading stuff under the radar
> without being asked (etc.). Stopping that can be a pain & I always
> groan whenever some package insists on using pip. Maybe I don't
> understand it well enough but in this role its dependency handling
> is an unnecessary complication with no purpose.

There are two cases where a 'pip install' run might go off and start
downloading packages without asking you:

- if the project is using setuptools with setup_requires=..., then the
setup.py itself will go off and start downloading things without asking.
This has nothing to do with pip. The way Debian prevents this is that they
always define an intentionally invalid http_proxy environment variable
before building any python package.

- if the project has declared that they do not work without some other
package installed via install_requires=... For this case, if you really
know what you're doing and you intentionally want to install a
non-functional configuration (which yeah, a package build tool might indeed
want to do), then just add --no-deps to the pip install command line. Maybe
add --no-index and/or the magic http_proxy setting if you want to be extra
sure.

> Just a comment that
> not every installation is someone trying to get numpy on their
> laptop...

Sure, we're well aware of the importance of downstream packagers -- part of
the point of having this email thread is to smoke out such non-trivial use
cases. (And note that worst case if you decide that you'd rather take your
chances with setup.py install, then that's why the proposal includes an
escape hatch of passing a special --force switch.)

But unless you're somehow planning to disable pip entirely in your
distribution, so that end users have to get upgrades through your tools
rather than using pip, then you do probably want to think about how to
provide accurate pip-style metadata. (And even then it doesn't hurt.)

-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Nathan Goldbaum 
Interestingly, conda actually does "setup.py install" in the recipe for
numpy:
https://github.com/conda/conda-recipes/blob/master/numpy-openblas/build.sh

I'm not sure if this is the one they use to build the anaconda package, I
think they have internal versions of most of the recipes on conda-recipes.

On Tue, Oct 27, 2015 at 10:18 AM, Nathaniel Smith  wrote:

> On Oct 27, 2015 6:48 AM, "James E.H. Turner"  wrote:
> >>
> >> Apparently it is not well known that if you have a Python project
> >> source tree (e.g., a numpy checkout), then the correct way to install
> >> it is NOT to type
> >>
> >>python setup.py install   # bad and broken!
> >>
> >> but rather to type
> >>
> >>pip install .
> >
> >
> > Though I haven't studied it exhaustively, it always seems to me that
> > pip is bad & broken, whereas python setup.py install does what I
> > expect (even if it's a mess internally).
>
> Unfortunately this is only true if what you expect is for packages to be
> installed in subtly corrupted ways, as described in the original email.
> Sorry to be the bearer of bad tidings :-/
>
> > In particular, when
> > maintaining a distribution of Python packages, you try to have some
> > well-defined, reproducible build from source tarballs and then you
> > find that pip is going off and downloading stuff under the radar
> > without being asked (etc.). Stopping that can be a pain & I always
> > groan whenever some package insists on using pip. Maybe I don't
> > understand it well enough but in this role its dependency handling
> > is an unnecessary complication with no purpose.
>
> There are two cases where a 'pip install' run might go off and start
> downloading packages without asking you:
>
> - if the project is using setuptools with setup_requires=..., then the
> setup.py itself will go off and start downloading things without asking.
> This has nothing to do with pip. The way Debian prevents this is that they
> always define an intentionally invalid http_proxy environment variable
> before building any python package.
>
> - if the project has declared that they do not work without some other
> package installed via install_requires=... For this case, if you really
> know what you're doing and you intentionally want to install a
> non-functional configuration (which yeah, a package build tool might indeed
> want to do), then just add --no-deps to the pip install command line. Maybe
> add --no-index and/or the magic http_proxy setting if you want to be extra
> sure.
>
> > Just a comment that
> > not every installation is someone trying to get numpy on their
> > laptop...
>
> Sure, we're well aware of the importance of downstream packagers -- part
> of the point of having this email thread is to smoke out such non-trivial
> use cases. (And note that worst case if you decide that you'd rather take
> your chances with setup.py install, then that's why the proposal includes
> an escape hatch of passing a special --force switch.)
>
> But unless you're somehow planning to disable pip entirely in your
> distribution, so that end users have to get upgrades through your tools
> rather than using pip, then you do probably want to think about how to
> provide accurate pip-style metadata. (And even then it doesn't hurt.)
>
> -n
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Benjamin Root 
Conda is for binary installs and largely targeted for end-users. This topic
pertains to source installs, and is mostly relevant to developers, testers,
and those who like to live on the bleeding edge of a particular project.

On Tue, Oct 27, 2015 at 10:31 AM, Edison Gustavo Muenz <
edisongust...@gmail.com> wrote:

> I'm sorry if this is out-of-topic, but I'm curious on why nobody mentioned
> Conda yet.
>
> Is there any particular reason for not using it?
>
> On Tue, Oct 27, 2015 at 11:48 AM, James E.H. Turner 
> wrote:
>
>> Apparently it is not well known that if you have a Python project
>>> source tree (e.g., a numpy checkout), then the correct way to install
>>> it is NOT to type
>>>
>>>python setup.py install   # bad and broken!
>>>
>>> but rather to type
>>>
>>>pip install .
>>>
>>
>> Though I haven't studied it exhaustively, it always seems to me that
>> pip is bad & broken, whereas python setup.py install does what I
>> expect (even if it's a mess internally). In particular, when
>> maintaining a distribution of Python packages, you try to have some
>> well-defined, reproducible build from source tarballs and then you
>> find that pip is going off and downloading stuff under the radar
>> without being asked (etc.). Stopping that can be a pain & I always
>> groan whenever some package insists on using pip. Maybe I don't
>> understand it well enough but in this role its dependency handling
>> is an unnecessary complication with no purpose. Just a comment that
>> not every installation is someone trying to get numpy on their
>> laptop...
>>
>> Cheers,
>>
>> James.
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread David Cournapeau 
On Tue, Oct 27, 2015 at 2:31 PM, Edison Gustavo Muenz <
edisongust...@gmail.com> wrote:

> I'm sorry if this is out-of-topic, but I'm curious on why nobody mentioned
> Conda yet.
>

Conda is a binary distribution system, whereas we are talking about
installing from sources. You will need a way to install things when
building a conda package in any case

David


> Is there any particular reason for not using it?
>
> On Tue, Oct 27, 2015 at 11:48 AM, James E.H. Turner 
> wrote:
>
>> Apparently it is not well known that if you have a Python project
>>> source tree (e.g., a numpy checkout), then the correct way to install
>>> it is NOT to type
>>>
>>>python setup.py install   # bad and broken!
>>>
>>> but rather to type
>>>
>>>pip install .
>>>
>>
>> Though I haven't studied it exhaustively, it always seems to me that
>> pip is bad & broken, whereas python setup.py install does what I
>> expect (even if it's a mess internally). In particular, when
>> maintaining a distribution of Python packages, you try to have some
>> well-defined, reproducible build from source tarballs and then you
>> find that pip is going off and downloading stuff under the radar
>> without being asked (etc.). Stopping that can be a pain & I always
>> groan whenever some package insists on using pip. Maybe I don't
>> understand it well enough but in this role its dependency handling
>> is an unnecessary complication with no purpose. Just a comment that
>> not every installation is someone trying to get numpy on their
>> laptop...
>>
>> Cheers,
>>
>> James.
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Nathaniel Smith 
On Oct 27, 2015 8:34 AM, "Nathan Goldbaum"  wrote:
>
> Would this happen at the level of numpy's setup.py script or would it be
implemented in numpy.distutils? I'm asking as the developer of a package
that uses numpy.distutils to manage C extensions.

NumPy's setup.py, no effect on numpy.distutils users. Unless you also get
fed up and implement the same thing, of course ;-).

-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Juan Nunez-Iglesias 
Can someone here who understands more about distribution maybe write a blog 
post detailing:




- why these setup.py commands are bad

- which alternative corresponds to each command and why it's better

- where to find information about this




For example, I had never heard of "twine", and parenthetical statements such as 
"setup.py upload (which is broken and should never be used)" are useless to 
those who don't know this and useless to those who do.





I understand that this is an "internal" discussion, but it's nice if those 
following to learn can get quick pointers. Since there is a *ton* of material 
online telling us *to use* python setup.py install, all the time, it would be 
extremely helpful for the community if discussions such as this one helped to 
bubble up the Right Way of doing Python packaging and distribution.




Thanks,




Juan.

On Wed, Oct 28, 2015 at 9:16 AM, Ralf Gommers 
wrote:

> On Tue, Oct 27, 2015 at 8:19 AM, Ralf Gommers 
> wrote:
> Updating this list for comments made after I sent it and now that I've
> looked in more detail at what the less common commands do:
>> So if/when we accept the proposal in this thread, I'm thinking we should
>> make a bunch of changes at once:
>> - always use setuptools (this is a new dependency)
>> - error on ``python setup.py install``
>>
> (removed the item about setup_requires, relevant for scipy but not numpy)
>> - error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
>> -Xdf ...) instead")
>> - change ``python setup.py --help`` to first show numpy-specific stuff
>> before setuptools help info
>> - update all our install docs
>>
> - error on ``python setup.py upload`` (saying "use `twine upload -s`
> instead")
> - error on ``python setup.py upload_docs``
> - error on ``python setup.py easy_install`` (I'm not joking, that exists)
> - error on ``python setup.py test`` (saying "use `python runtests.py`
> instead")
> - remove setupegg.py
> Ralf
> And when "pip upgrade" is released (should be soon, see
>> https://github.com/pypa/pip/pull/3194), officially change our mind and
>> recommend the use of install_requires/setup_requires to packages depending
>> on numpy.
>>
>>___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Ralf Gommers 
On Tue, Oct 27, 2015 at 8:19 AM, Ralf Gommers 
wrote:

Updating this list for comments made after I sent it and now that I've
looked in more detail at what the less common commands do:


> So if/when we accept the proposal in this thread, I'm thinking we should
> make a bunch of changes at once:
> - always use setuptools (this is a new dependency)
> - error on ``python setup.py install``
>

(removed the item about setup_requires, relevant for scipy but not numpy)


> - error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
> -Xdf ...) instead")
> - change ``python setup.py --help`` to first show numpy-specific stuff
> before setuptools help info
> - update all our install docs
>

- error on ``python setup.py upload`` (saying "use `twine upload -s`
instead")
- error on ``python setup.py upload_docs``
- error on ``python setup.py easy_install`` (I'm not joking, that exists)
- error on ``python setup.py test`` (saying "use `python runtests.py`
instead")
- remove setupegg.py

Ralf

And when "pip upgrade" is released (should be soon, see
> https://github.com/pypa/pip/pull/3194), officially change our mind and
> recommend the use of install_requires/setup_requires to packages depending
> on numpy.
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Ralf Gommers 
On Tue, Oct 27, 2015 at 4:28 PM,  wrote:

>
>
>
> On Tue, Oct 27, 2015 at 10:59 AM, Nathaniel Smith  wrote:
>
>> On Oct 27, 2015 6:08 AM,  wrote:
>> >
>> [...]
>> >
>> >
>> > What's the equivalent of
>> > python setup.py build_ext --inplace
>>
>> It's
>> python setup.py build_ext --inplace
>>
>> ;-)
>>
> Ok, Sorry, I read now the small print and the issue.
>
> Sounds reasonable, given we can `force` our way out.
>
> (If the reason to run to pip is a misspelled dev version number, then it
> looks like a hammer to me.)
>

That's not the reason. A main reason is that we want reliable uninstall of
numpy, as explained in the initial post. We also want to avoid as many
other broken parts of setuptools/easy_install as possible.

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Juan Nunez-Iglesias 
Thanks Ralf! The pointer to Python Packaging User Guide is already gold! (But a 
wider discussion e.g. in the NumPy repo, mirroring the docstring conventions, 
would also be good!)

On Wed, Oct 28, 2015 at 10:02 AM, Ralf Gommers 
wrote:

> On Tue, Oct 27, 2015 at 11:35 PM, Juan Nunez-Iglesias 
> wrote:
>> Can someone here who understands more about distribution maybe write a
>> blog post detailing:
>>
>> - why these setup.py commands are bad
>> - which alternative corresponds to each command and why it's better
>> - where to find information about this
>>
> Good question. Not that I have a blog, but I can try to write something a
> bit longer the coming weekend.
>>
>> For example, I had never heard of "twine", and parenthetical statements
>> such as "setup.py upload (which is broken and should never be used)" are
>> useless to those who don't know this and useless to those who do.
>>
> IIRC `setup.py upload` sends passwords over plain http. I've also seen it
> do weird things like change one's own PyPi rights from maintainer to owner.
> The most comprehensive overview of all this stuff is
> https://packaging.python.org/en/latest/, which starts with tool
> recommendations. Twine is one of the first things mentioned.
> Ralf
>> I understand that this is an "internal" discussion, but it's nice if those
>> following to learn can get quick pointers. Since there is a *ton* of
>> material online telling us *to use* python setup.py install, all the time,
>> it would be extremely helpful for the community if discussions such as this
>> one helped to bubble up the Right Way of doing Python packaging and
>> distribution.
>>
>> Thanks,
>>
>> Juan.
>>
>>
>>
>>
>>
>> On Wed, Oct 28, 2015 at 9:16 AM, Ralf Gommers 
>> wrote:
>>
>>>
>>>
>>> On Tue, Oct 27, 2015 at 8:19 AM, Ralf Gommers 
>>> wrote:
>>>
>>> Updating this list for comments made after I sent it and now that I've
>>> looked in more detail at what the less common commands do:
>>>
>>>
 So if/when we accept the proposal in this thread, I'm thinking we should
 make a bunch of changes at once:
 - always use setuptools (this is a new dependency)
 - error on ``python setup.py install``

>>>
>>> (removed the item about setup_requires, relevant for scipy but not numpy)
>>>
>>>
 - error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
 -Xdf ...) instead")
 - change ``python setup.py --help`` to first show numpy-specific stuff
 before setuptools help info
 - update all our install docs

>>>
>>> - error on ``python setup.py upload`` (saying "use `twine upload -s`
>>> instead")
>>> - error on ``python setup.py upload_docs``
>>> - error on ``python setup.py easy_install`` (I'm not joking, that exists)
>>> - error on ``python setup.py test`` (saying "use `python runtests.py`
>>> instead")
>>> - remove setupegg.py
>>>
>>> Ralf
>>>
>>> And when "pip upgrade" is released (should be soon, see
 https://github.com/pypa/pip/pull/3194), officially change our mind and
 recommend the use of install_requires/setup_requires to packages depending
 on numpy.


>>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Ralf Gommers 
On Tue, Oct 27, 2015 at 11:35 PM, Juan Nunez-Iglesias 
wrote:

> Can someone here who understands more about distribution maybe write a
> blog post detailing:
>
> - why these setup.py commands are bad
> - which alternative corresponds to each command and why it's better
> - where to find information about this
>

Good question. Not that I have a blog, but I can try to write something a
bit longer the coming weekend.


>
> For example, I had never heard of "twine", and parenthetical statements
> such as "setup.py upload (which is broken and should never be used)" are
> useless to those who don't know this and useless to those who do.
>

IIRC `setup.py upload` sends passwords over plain http. I've also seen it
do weird things like change one's own PyPi rights from maintainer to owner.

The most comprehensive overview of all this stuff is
https://packaging.python.org/en/latest/, which starts with tool
recommendations. Twine is one of the first things mentioned.

Ralf



> I understand that this is an "internal" discussion, but it's nice if those
> following to learn can get quick pointers. Since there is a *ton* of
> material online telling us *to use* python setup.py install, all the time,
> it would be extremely helpful for the community if discussions such as this
> one helped to bubble up the Right Way of doing Python packaging and
> distribution.
>
> Thanks,
>
> Juan.
>
>
>
>
>
> On Wed, Oct 28, 2015 at 9:16 AM, Ralf Gommers 
> wrote:
>
>>
>>
>> On Tue, Oct 27, 2015 at 8:19 AM, Ralf Gommers 
>> wrote:
>>
>> Updating this list for comments made after I sent it and now that I've
>> looked in more detail at what the less common commands do:
>>
>>
>>> So if/when we accept the proposal in this thread, I'm thinking we should
>>> make a bunch of changes at once:
>>> - always use setuptools (this is a new dependency)
>>> - error on ``python setup.py install``
>>>
>>
>> (removed the item about setup_requires, relevant for scipy but not numpy)
>>
>>
>>> - error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
>>> -Xdf ...) instead")
>>> - change ``python setup.py --help`` to first show numpy-specific stuff
>>> before setuptools help info
>>> - update all our install docs
>>>
>>
>> - error on ``python setup.py upload`` (saying "use `twine upload -s`
>> instead")
>> - error on ``python setup.py upload_docs``
>> - error on ``python setup.py easy_install`` (I'm not joking, that exists)
>> - error on ``python setup.py test`` (saying "use `python runtests.py`
>> instead")
>> - remove setupegg.py
>>
>> Ralf
>>
>> And when "pip upgrade" is released (should be soon, see
>>> https://github.com/pypa/pip/pull/3194), officially change our mind and
>>> recommend the use of install_requires/setup_requires to packages depending
>>> on numpy.
>>>
>>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread Edison Gustavo Muenz 
I'm sorry if this is out-of-topic, but I'm curious on why nobody mentioned
Conda yet.

Is there any particular reason for not using it?

On Tue, Oct 27, 2015 at 11:48 AM, James E.H. Turner 
wrote:

> Apparently it is not well known that if you have a Python project
>> source tree (e.g., a numpy checkout), then the correct way to install
>> it is NOT to type
>>
>>python setup.py install   # bad and broken!
>>
>> but rather to type
>>
>>pip install .
>>
>
> Though I haven't studied it exhaustively, it always seems to me that
> pip is bad & broken, whereas python setup.py install does what I
> expect (even if it's a mess internally). In particular, when
> maintaining a distribution of Python packages, you try to have some
> well-defined, reproducible build from source tarballs and then you
> find that pip is going off and downloading stuff under the radar
> without being asked (etc.). Stopping that can be a pain & I always
> groan whenever some package insists on using pip. Maybe I don't
> understand it well enough but in this role its dependency handling
> is an unnecessary complication with no purpose. Just a comment that
> not every installation is someone trying to get numpy on their
> laptop...
>
> Cheers,
>
> James.
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-27 Thread James E.H. Turner 

Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type

   python setup.py install   # bad and broken!

but rather to type

   pip install .


Though I haven't studied it exhaustively, it always seems to me that
pip is bad & broken, whereas python setup.py install does what I
expect (even if it's a mess internally). In particular, when
maintaining a distribution of Python packages, you try to have some
well-defined, reproducible build from source tarballs and then you
find that pip is going off and downloading stuff under the radar
without being asked (etc.). Stopping that can be a pain & I always
groan whenever some package insists on using pip. Maybe I don't
understand it well enough but in this role its dependency handling
is an unnecessary complication with no purpose. Just a comment that
not every installation is someone trying to get numpy on their
laptop...

Cheers,

James.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-26 Thread Nathaniel Smith 
On Mon, Oct 26, 2015 at 9:31 PM, Nathaniel Smith  wrote:
[...]
> I believe that this would also break both 'easy_install numpy', and
> attempts to install numpy via the setup_requires= argument to
> setuptools.setup (because setup_requires= implicitly calls
> easy_install). install_requires= would *not* be affected, and
> setup_requires= would still be fine in cases where numpy was already
> installed.

On further investigation, it looks like the simplest approach to doing
this would actually treat easy_install and setup_requires= the same
way as they treat pip, i.e., they would all be allowed. (I was
misreading some particularly confusing code in setuptools.)

It also looks like easy_installed packages can at least be safely
upgraded, so I guess allowing this is okay :-).

-n

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

2015-10-26 Thread Nathaniel Smith 
Hi all,

Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type

  python setup.py install   # bad and broken!

but rather to type

  pip install .

(I.e., pip install isn't just for packages on pypi -- you can also
pass it the path to an arbitrary source directory or the URL of a
source tarball and it will do its thing. In this case "install ."
means "install the project in the current directory".)

These don't quite have identical results -- the main difference is
that the latter makes sure that proper metadata gets installed so that
later on it will be possible to upgrade or uninstall correctly. If you
call setup.py directly, and then later you try to upgrade your
package, then it's entirely possible to end up with a mixture of old
and new versions of the package installed in your PYTHONPATH. (One
common effect is in numpy's case is that we get people sending us
mysterious bug reports about failing tests in files don't even exist
(!) -- because nose is finding tests in files from one version of
numpy and running them against a different version of numpy.)

But this isn't the only issue -- using pip also avoids a bunch of
weird corner cases in distutils/setuptools. E.g., if setup.py uses
plain distutils, then it turns out this will mangle numpy version
numbers in ways that cause weird horribleness -- see [1] for a bug
report of the form "matplotlib doesn't build anymore" which turned out
to be because of using 'setup.py install' to install numpy. OTOH if
setup.py uses setuptools then you get different weirdnesses, like you
can easily end up with multiple versions of the same library installed
simultaneously.

And finally, an advantage of getting used to using 'pip install .' now
is that you'll be prepared for the glorious future when we kill
distutils and get rid of setup.py entirely in favor of something less
terrible [2].

So a proposal that came out of the discussion in [1] is that we modify
numpy's setup.py now so that if you try running

python setup.py install

you get

Error: Calling 'setup.py install' directly is NOT SUPPORTED!
Instead, do:

pip install .

Alternatively, if you want to proceed at your own risk, you
can try 'setup.py install --force-raw-setup.py'
For more information see http://...

(Other setup.py commands would continue to work as normal.)

I believe that this would also break both 'easy_install numpy', and
attempts to install numpy via the setup_requires= argument to
setuptools.setup (because setup_requires= implicitly calls
easy_install). install_requires= would *not* be affected, and
setup_requires= would still be fine in cases where numpy was already
installed.

This would hopefully cut down on the amount of time everyone spends
trying to track down these stupid weird bugs, but it will also require
some adjustment in people's workflows, so... objections? concerns?

-n

[1] https://github.com/numpy/numpy/issues/6551
[2] https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html

-- 
Nathaniel J. Smith -- http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion