Re: [Distutils] name of the dependency problem

2015-04-15 Thread Justin Cappos
Yes, it's another way to solve the problem. Both backtracking dependency resolution and ZYpp will always find a solution. The tradeoff is really in how they function. ZYpp is faster if there are a lot of dependencies that conflict. The backtracking dependency resolution used in Stork is much

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Trishank Karthik Kuppusamy
On 4/15/15 9:28 AM, Justin Cappos wrote: Yes, it's another way to solve the problem. Both backtracking dependency resolution and ZYpp will always find a solution. The tradeoff is really in how they function. ZYpp is faster if there are a lot of dependencies that conflict. The backtracking

Re: [Distutils] buildout/setuptools slow as it scans the whole project dir

2015-04-15 Thread Jim Fulton
On Wed, Apr 15, 2015 at 10:01 AM, Reinout van Rees rein...@vanrees.org wrote: Hi, In some of my projects, buildout takes a looong time to complete. Sometimes the same problem occurs on the server or on another developer's laptop, sometimes not. The difference is something like 15 seconds

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Daniel Holth
See also http://en.wikipedia.org/wiki/ZYpp On Wed, Apr 15, 2015 at 7:43 AM, Justin Cappos jcap...@nyu.edu wrote: First of all, I'm surprised that pip doesn't warn or error in this case. I think this is certainly a bug that should be fixed. The problem can come up in much more subtle cases

[Distutils] buildout/setuptools slow as it scans the whole project dir

2015-04-15 Thread Reinout van Rees
Hi, In some of my projects, buildout takes a looong time to complete. Sometimes the same problem occurs on the server or on another developer's laptop, sometimes not. The difference is something like 15 seconds if there's no problem and 5 minutes if the problem occurs. It is terribly

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Robin Becker
Thanks to Justin Daniel for some insights. On 15/04/2015 14:28, Justin Cappos wrote: Yes, it's another way to solve the problem. Both backtracking dependency resolution and ZYpp will always find a solution. The tradeoff is really in how they function. ZYpp is faster if there are a lot of

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Justin Cappos
I guess I should provide the code for what I've done in this space also. Yes, the problem is NP hard, so in both cases, the system may run really slowly. In practice, you don't tend to have a lot of conflicts you need to resolve during a package install though. So I'd argue that optimizing for

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Marcus Smith
level 0 A A level 1 1.4= C level 0 B B level 1 1.6= C 1.7 pip manages to download version 1.8 of C(Django) using A's requirement, but never even warns us that the B requirement of C was violated. Surely even in the absence of a resolution pip could raise a warning at the end.

Re: [Distutils] buildout/setuptools slow as it scans the whole project dir

2015-04-15 Thread Reinout van Rees
Reinout van Rees schreef op 15-04-15 om 16:26: Setuptools seems to run all egg_info.writers entry points it can find in the piece of code where the slowness occurs. So any of the entry points could be the culprit. And perhaps even an entry point outside of setuptools, due to the way entry

Re: [Distutils] buildout/setuptools slow as it scans the whole project dir

2015-04-15 Thread Reinout van Rees
Jim Fulton schreef op 15-04-15 om 16:14: I wonder if the culprit is _dir_hash in zc.buildout.buildlout. Buildout reinstalls a part if it has changed. It considers a part to have changed if it's arguments have changed or it's recipe has changed. If a recipe is a develop egg, then it computes a

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Robin Becker
On 15/04/2015 14:34, Trishank Karthik Kuppusamy wrote: On 4/15/15 9:28 AM, Justin Cappos wrote: . ZYpp seems to assume that dependency resolution is an NP-complete problem (http://www.watzmann.net/blog/2005/11/package-installation-is-np-complete.html). yes I deduced that this must be

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Robin Becker
On 15/04/2015 16:49, Marcus Smith wrote: .. agreed on the warning, but there is a documented workaround for this, that is to put the desired constraint for C at level 0 (i.e. in the install args or in your requirements file) see case #2 here

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Trishank Karthik Kuppusamy
On 15 April 2015 at 11:15, David Cournapeau courn...@gmail.com wrote: This is indeed the case. If you want to solve dependencies in a way that works well, you want an index that describes all your available package versions. While solving dependencies is indeed NP complete, they can be

Re: [Distutils] buildout/setuptools slow as it scans the whole project dir

2015-04-15 Thread Ionel Cristian Mărieș
Could this be the same problem https://bitbucket.org/pypa/setuptools/issue/249/have-a-way-to-ingore-specific-dirs-when ? I frequently have this issue (.tox dir can sometimes get quite large) and I resorted to monkeypatching setuptools (I know, right?) in

Re: [Distutils] buildout/setuptools slow as it scans the whole project dir

2015-04-15 Thread Reinout van Rees
Reinout van Rees schreef op 15-04-15 om 17:01: So it almost seems as if there's no solution to this? Or can someone give a hint on os.walk performance relative to VMs? Bingo! I think I've found the real problem: symlinks between filesystems. The project is on the host file system and is

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Marcus Smith
On Wed, Apr 15, 2015 at 8:56 AM, Robin Becker ro...@reportlab.com wrote: On 15/04/2015 16:49, Marcus Smith wrote: .. agreed on the warning, but there is a documented workaround for this, that is to put the desired constraint for C at level 0 (i.e. in the install args or in your

Re: [Distutils] name of the dependency problem

2015-04-15 Thread David Cournapeau
On Wed, Apr 15, 2015 at 11:32 AM, Trishank Karthik Kuppusamy trish...@nyu.edu wrote: On 15 April 2015 at 11:15, David Cournapeau courn...@gmail.com wrote: This is indeed the case. If you want to solve dependencies in a way that works well, you want an index that describes all your available

Re: [Distutils] buildout/setuptools slow as it scans the whole project dir

2015-04-15 Thread Reinout van Rees
Ionel Cristian Mărieș schreef op 15-04-15 om 19:04: Could this be the same problem https://bitbucket.org/pypa/setuptools/issue/249/have-a-way-to-ingore-specific-dirs-when ? Yes, that's the same. Though: normally reading such a dir structure shouldn't take long. See my other answer, in my

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

2015-04-15 Thread Chris Barker
On Tue, Apr 14, 2015 at 8:57 PM, Kevin Horn kevin.h...@gmail.com wrote: Personally, I'm not a fan of auto-installing, I'm with Paul on this one. It seems to me that auto-installing the extension would destroy most of the advantages of distributing the extensions separately. Exactly -- I

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Jeremy Stanley
On 2015-04-15 12:09:04 +0100 (+0100), Robin Becker wrote: After again finding that pip doesn't have a correct dependency resolution solution a colleage and I discussed the nature of the problem. [...] Before the discussion of possible solutions heads too far afield, it's worth noting that this

Re: [Distutils] version for VCS/local builds between alpha and release

2015-04-15 Thread Nick Coghlan
On 14 Apr 2015 04:04, Robert Collins robe...@robertcollins.net wrote: Tl;dr: I woud like to change PEP-440 to remove the stigmata around dev versions that come between pre-releases and releases. The basic scenario here is developers and CD deployers building versions from VCS of arbitrary

Re: [Distutils] version for VCS/local builds between alpha and release

2015-04-15 Thread Robert Collins
Thanks. There's still a little too much negative there for me - if I propose a diff here would that be ok? On 16 April 2015 at 11:26, Nick Coghlan ncogh...@gmail.com wrote: On 14 Apr 2015 04:04, Robert Collins robe...@robertcollins.net wrote: Tl;dr: I woud like to change PEP-440 to remove the

Re: [Distutils] version for VCS/local builds between alpha and release

2015-04-15 Thread Nick Coghlan
On 15 April 2015 at 19:33, Robert Collins robe...@robertcollins.net wrote: Thanks. There's still a little too much negative there for me - if I propose a diff here would that be ok? Sure, or a PR against the GitHub mirror:

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

2015-04-15 Thread Steve Dower
On the Start Menu suggestion, I think that's a horrible idea. Pip is not the system package manager and it shouldn't be changing the system. Unversioned script launchers are in the same category, but aren't quite as offensive. I know it's only a hypothetical, but I'd much rather it didn't get

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Robert Collins
On 16 April 2015 at 04:52, David Cournapeau courn...@gmail.com wrote: On Wed, Apr 15, 2015 at 11:32 AM, Trishank Karthik Kuppusamy trish...@nyu.edu wrote: On 15 April 2015 at 11:15, David Cournapeau courn...@gmail.com wrote: This is indeed the case. If you want to solve dependencies in a

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Wes Turner
IIRC, Conda (BSD License) takes a SAT solving (e.g. Sudoku) approach: http://continuum.io/blog/new-advances-in-conda (such as installing pycosat (MIT License) when I install conda). Some links to the source: * https://github.com/conda/conda/blob/master/conda/logic.py *

Re: [Distutils] Beyond wheels 1.0: helping downstream, FHS and more

2015-04-15 Thread Paul Moore
On 15 April 2015 at 21:40, Chris Barker chris.bar...@noaa.gov wrote: Which brings us back to the review of extensions thing -- I think it's less about the end user checking it out and making a decision about it, but about the package builder doing that. I have a package I want to be easy to

[Distutils] name of the dependency problem

2015-04-15 Thread Robin Becker
After again finding that pip doesn't have a correct dependency resolution solution a colleage and I discussed the nature of the problem. We examined the script capture of our install and it seems as though when presented with level 0 A A level 1 1.4= C level 0 B B level 1 1.6= C 1.7

Re: [Distutils] name of the dependency problem

2015-04-15 Thread Justin Cappos
First of all, I'm surprised that pip doesn't warn or error in this case. I think this is certainly a bug that should be fixed. The problem can come up in much more subtle cases too that are very hard for the user to understand. The good news is that this is a known problem that happens when