Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-08 Thread Jeremy Stanley
On 2015-06-09 05:49:35 +1200 (+1200), Robert Collins wrote:
> I already said that the git entry should be to a local zuul-cloner
> cloned repo. Kevin's *current* 3rd-party CI solution is doing
> full-clones each time.

Aah, yep, I missed in his reply that it would be a local repo on the
filesystem referenced as a URL in test-requirements.txt. Something
still needs to create that local repo, so for devs running tox
locally a little additional automation is needed. Sounds like that's
what his installer script callout would take care of.

> BTW pip can trivially re-use git repositories if you specify the
> --source path in e.g. pip.conf, but thats a separate discussion IMO
> :).

Oh! That's useful. I wonder if we could leverage that in our CI to
tell tox where to find git repos we've pre-cloned for the job
without having to precreate an env and manually pip install into it.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-08 Thread Robert Collins
On 9 June 2015 at 04:19, Doug Wiegley  wrote:
>
>> On Jun 8, 2015, at 9:58 AM, Jeremy Stanley  wrote:
>>
>> On 2015-06-08 13:29:50 +1200 (+1200), Robert Collins wrote:
>> [...]
>>> However, we are going to move from test-requirements.txt to setup.cfg
>>> eventually, but thats a separate transition - and one could still use
>>> test-requirements.txt there to provide git references.
>>
>> Except please don't. If you put a git requirement entry in that file
>> for neutron, every CI job which uses that test-requirements.txt will
>> reclone all of neutron over the network from scratch because pip
>> isn't smart enough to do otherwise unless you take additional
>> measures to preinstall that requirement in the environment where the
>> test is run. That's why we use tools like devstack-gate or
>> zuul-cloner which know how to check for cached repos and update them to
>> the ref that you've (or that zuul has) requested.
>
> The neutron-*aas repos were among the worst offenders in the ‘ninja cloning’ 
> racket, and in addition to working against the CI infrastructure, it bit us 
> by testing master instead of the review patches in some cases. Note that we 
> were grabbing neutron at master and tempest at a pinned commit. And since we 
> use the same tox env’s for devs and CI, there are slightly different 
> requirements depending on which is in use.

tempest-lib or tempest? tempest-lib cuts releases and should AIUI be
consumed via those: if you need something thats not in the latest
release, ask that it be released.

> What I am attempting to do, and I’m open to feedback on ‘best practices’:
>
> - pep8/unit tests - override ‘install_program’ in tox, to be an in-repo 
> script that detects if you’re inside jenkins or not, and then uses either 
> zuul-cloner or pulls in neutron via a git url, respectively, then calls ‘pip 
> install’ as usual for the rest.

So, we're going to be adding in this thing for constraints over this
cycle (https://review.openstack.org/#/c/186635/) and it will also be
gluing into tox in that sort of way. Since you can specify a git url
and reference there, I wonder if editing the constraints file
just-in-time to add e.g. a neutron reference, will be useful. Or yeah
zuul-cloner. We'll need to make sure these things play nice together.

> - tempest api tests - pulled in-tree the subset of tempest that we’re using 
> and which is not yet in tempest-lib, with an eye to migrating it away. This 
> part could’ve also been done via zuul-cloner at a specific commit.

I'd get the stuff into tempest-lib ASAP rather than re-arranging the
way this is done IMO.

> - We do *not* put ‘neutron’ into any requirements file, and I wasn’t planning 
> to, until such time as neutron, or neutron-lib, is stable and published to 
> pypi.

Without neutron there, there is no signal to packagers, deployers, or
pip that neutron is needed. I think thats just bugs waiting to happen.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-08 Thread Robert Collins
On 9 June 2015 at 03:58, Jeremy Stanley  wrote:
> On 2015-06-08 13:29:50 +1200 (+1200), Robert Collins wrote:
> [...]
>> However, we are going to move from test-requirements.txt to setup.cfg
>> eventually, but thats a separate transition - and one could still use
>> test-requirements.txt there to provide git references.
>
> Except please don't. If you put a git requirement entry in that file
> for neutron, every CI job which uses that test-requirements.txt will
> reclone all of neutron over the network from scratch because pip
> isn't smart enough to do otherwise unless you take additional
> measures to preinstall that requirement in the environment where the
> test is run. That's why we use tools like devstack-gate or
> zuul-cloner which know how to check for cached repos and update them to
> the ref that you've (or that zuul has) requested.

I already said that the git entry should be to a local zuul-cloner
cloned repo. Kevin's *current* 3rd-party CI solution is doing
full-clones each time.

BTW pip can trivially re-use git repositories if you specify the
--source path in e.g. pip.conf, but thats a separate discussion IMO
:).

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-08 Thread Doug Wiegley

> On Jun 8, 2015, at 9:58 AM, Jeremy Stanley  wrote:
> 
> On 2015-06-08 13:29:50 +1200 (+1200), Robert Collins wrote:
> [...]
>> However, we are going to move from test-requirements.txt to setup.cfg
>> eventually, but thats a separate transition - and one could still use
>> test-requirements.txt there to provide git references.
> 
> Except please don't. If you put a git requirement entry in that file
> for neutron, every CI job which uses that test-requirements.txt will
> reclone all of neutron over the network from scratch because pip
> isn't smart enough to do otherwise unless you take additional
> measures to preinstall that requirement in the environment where the
> test is run. That's why we use tools like devstack-gate or
> zuul-cloner which know how to check for cached repos and update them to
> the ref that you've (or that zuul has) requested.

The neutron-*aas repos were among the worst offenders in the ‘ninja cloning’ 
racket, and in addition to working against the CI infrastructure, it bit us by 
testing master instead of the review patches in some cases. Note that we were 
grabbing neutron at master and tempest at a pinned commit. And since we use the 
same tox env’s for devs and CI, there are slightly different requirements 
depending on which is in use.

What I am attempting to do, and I’m open to feedback on ‘best practices’:

- pep8/unit tests - override ‘install_program’ in tox, to be an in-repo script 
that detects if you’re inside jenkins or not, and then uses either zuul-cloner 
or pulls in neutron via a git url, respectively, then calls ‘pip install’ as 
usual for the rest.

- tempest api tests - pulled in-tree the subset of tempest that we’re using and 
which is not yet in tempest-lib, with an eye to migrating it away. This part 
could’ve also been done via zuul-cloner at a specific commit.

- We do *not* put ‘neutron’ into any requirements file, and I wasn’t planning 
to, until such time as neutron, or neutron-lib, is stable and published to pypi.

Thanks,
doug



> -- 
> Jeremy Stanley
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-08 Thread Jeremy Stanley
On 2015-06-08 13:29:50 +1200 (+1200), Robert Collins wrote:
[...]
> However, we are going to move from test-requirements.txt to setup.cfg
> eventually, but thats a separate transition - and one could still use
> test-requirements.txt there to provide git references.

Except please don't. If you put a git requirement entry in that file
for neutron, every CI job which uses that test-requirements.txt will
reclone all of neutron over the network from scratch because pip
isn't smart enough to do otherwise unless you take additional
measures to preinstall that requirement in the environment where the
test is run. That's why we use tools like devstack-gate or
zuul-cloner which know how to check for cached repos and update them to
the ref that you've (or that zuul has) requested.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-07 Thread Robert Collins
On 8 June 2015 at 11:16, Kevin Benton  wrote:
> It wasn't using zuul at all. It's a super short bash script that just clones
> the 3rd party repo, checks out the patch, and then runs 'tox -epy27'.
>
> I misspoke in my previous email, because it was setup to use
> test-requirements.txt to pull in neutron. Did I understand your other email
> that implied this break only affects requirements.txt?

Right. test-requirements.txt has never been processed by pbr, and
won't be. So that should be fine.

However, we are going to move from test-requirements.txt to setup.cfg
eventually, but thats a separate transition - and one could still use
test-requirements.txt there to provide git references.

-Rob



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-07 Thread Kevin Benton
It wasn't using zuul at all. It's a super short bash script that just
clones the 3rd party repo, checks out the patch, and then runs 'tox
-epy27'.

I misspoke in my previous email, because it was setup to use
test-requirements.txt to pull in neutron. Did I understand your other email
that implied this break only affects requirements.txt?
On Jun 7, 2015 4:10 PM, "Robert Collins"  wrote:

> On 5 June 2015 at 04:54, Kevin Benton  wrote:
> > +1. I had setup a CI for a third-party plugin and the easiest thing to
> do to
> > make sure it was running tests with the latest copy of the corresponding
> > neutron branch was to put the git URL in requirements.txt.
> >
> > We wanted to always test the latest code so we had early detection of
> > failures. What's the appropriate way to do that without using a git
> > reference?
>
> In the OpenStack gate, use the local cached Neutron git tree, updated
> by zuul-cloner. For third-party CI, it will depend on your test setup,
> how you're running tests, are you using images etc etc - but if its
> similar to the OpenStack gate, zuul-cloner should be the answer again.
>
> The only place its tricky is when dealing with tox environments, where
> you need to be able to inject something with the right local path - so
> a new file which you can reference with -r in tox.ini. (And -c in
> future, but that hasn't landed yet).
>
> -Rob
>
>
> --
> Robert Collins 
> Distinguished Technologist
> HP Converged Cloud
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-07 Thread Robert Collins
On 5 June 2015 at 04:54, Kevin Benton  wrote:
> +1. I had setup a CI for a third-party plugin and the easiest thing to do to
> make sure it was running tests with the latest copy of the corresponding
> neutron branch was to put the git URL in requirements.txt.
>
> We wanted to always test the latest code so we had early detection of
> failures. What's the appropriate way to do that without using a git
> reference?

In the OpenStack gate, use the local cached Neutron git tree, updated
by zuul-cloner. For third-party CI, it will depend on your test setup,
how you're running tests, are you using images etc etc - but if its
similar to the OpenStack gate, zuul-cloner should be the answer again.

The only place its tricky is when dealing with tox environments, where
you need to be able to inject something with the right local path - so
a new file which you can reference with -r in tox.ini. (And -c in
future, but that hasn't landed yet).

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-07 Thread Robert Collins
On 4 June 2015 at 21:06, Ihar Hrachyshka  wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> On 06/03/2015 11:08 PM, Robert Collins wrote:
...
>> One question that this raises, and this is why I wrote the email:
>> is there any need to support this at all:- can we say that we won't
>> use tarball/vcs support at all and block it as a policy step in
>> global requirements? AIUI both git and tarball support is
>> problematic for CI jobs due to the increased flakiness of depending
>> on network resources... so its actively harmful anyway.
>>
>
> Lots of Neutron modules, like advanced services, or out-of-tree
> plugins, rely on neutron code being checked out from git [1]. I don't
> say it's the way to go forward, and there were plans to stop relying
> on latest git to avoid frequent breakages, but it's not yet implemented.

So - testing with unreleased code for the servers is de rigeur in
OpenStack today. I'd actually like us to think about doing better, but
the current status is that we test latest nova against latest keystone
exclusively and so on.

Where we test against releases is where we use things as libraries.
So, if you're using Neutron as a library, I really would have expected
you to be using the stable interface etc etc, but - that ship has
sailed - hopefully it will improve over time.

To deliver a git dependency like that today, I recommend:
requirements.txt
=
neutron
=

And then use a separate requirements file in unit tests etc to specify
where to get neutron from.

requirements.txt is *special* because its interpreted by pbr as well
as being usable by pip - you just need a requirements file that isn't
interpreted by pbr. E.g. you could use dev-requirements.txt (which we
just removed from update.py :/ - we could add it back of course). Be
sure to use the local git path for the gate - nodes have neutron
pre-cached, and zuul-cloner can be used to get a local copy
appropriately.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-04 Thread Kevin Benton
+1. I had setup a CI for a third-party plugin and the easiest thing to do
to make sure it was running tests with the latest copy of the corresponding
neutron branch was to put the git URL in requirements.txt.

We wanted to always test the latest code so we had early detection of
failures. What's the appropriate way to do that without using a git
reference?

On Thu, Jun 4, 2015 at 2:06 AM, Ihar Hrachyshka  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> On 06/03/2015 11:08 PM, Robert Collins wrote:
> > Hi, right now there is a little used (e.g. its not in any active
> > project these days) previous feature of pbr/global-requirements:
> > we supported things that setuptools does not: to whit, tarball and
> > git requirements.
> >
> > Now, these things are supported by pip, so the implementation
> > involved recursing into pip from our setup.py (setup.py -> pbr ->
> > pip). What we exported into setuptools was only the metadata about
> > the dependency name. This meant that we were re-entering pip,
> > potentially many times - it was, to be blunt, awful.
> >
> > Fortunately we removed the recursive re-entry into pip in pbr 1.0.
> > This didn't remove the ability to parse requirements.txt files
> > that contain urls, but it does mean they are converted to the
> > simple dependency name when doing 'pip install .' in a project tree
> > (or pip install $projectname), and so they are effectively
> > unversioned - no lower and no upper bound. This works poorly in the
> > gate: please don't use tarball or git urls in requirements.txt (or
> > setup.cfg for that matter).
> >
> > We can still choose to use something from git or a tarball in test
> > jobs, *if* thats the right thing (which it rarely is: I'm just
> > being clear that the technical capability still exists)... but it
> > needs to be done outside of requirements.txt going forward. Its
> > also something that we can support with the new constraints system
> > if desired [which will operate globally once in place (it is an
> > extension of global-requirements)].
> >
> > One question that this raises, and this is why I wrote the email:
> > is there any need to support this at all:- can we say that we won't
> > use tarball/vcs support at all and block it as a policy step in
> > global requirements? AIUI both git and tarball support is
> > problematic for CI jobs due to the increased flakiness of depending
> > on network resources... so its actively harmful anyway.
> >
>
> Lots of Neutron modules, like advanced services, or out-of-tree
> plugins, rely on neutron code being checked out from git [1]. I don't
> say it's the way to go forward, and there were plans to stop relying
> on latest git to avoid frequent breakages, but it's not yet implemented.
>
> [1]:
> http://git.openstack.org/cgit/openstack/neutron-vpnaas/tree/tox.ini#n10
>
> Ihar
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2
>
> iQEcBAEBCAAGBQJVcBUnAAoJEC5aWaUY1u57N2EH/jUFd0H9pQ7LApSAIlDTEl2v
> WR1EXnc9Vxf5nCWq/qmncj3OCpMDlgL/ZMrFu74LRTDbe38+16kh+Fb+FvBEPGA4
> ZkQC3gyg22Se/QcerTxdPil16hnT912Hr3E0cTuu/4ktyipPrVsO39N56Jbrb6WQ
> SRCrEohIg7C3c0NgFcvBGh+S4rNf8IKT1oLzKrRhSLzIE8lSeGa1GNnSXPAXk19/
> 2KIEnqBz3Q5J6umTprB5DFdxMe93Pj6jZmGIMFaHXYgG/yTdKz3zzGM3hpuLyGUQ
> kKYEzFJZ4vf2c6NBg//GYTcAkGjkM2QmAnS+uoztU5vm4QRkLgGcDCz29eQ5ufA=
> =6bUu
> -END PGP SIGNATURE-
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kevin Benton
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-04 Thread Ihar Hrachyshka
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 06/03/2015 11:08 PM, Robert Collins wrote:
> Hi, right now there is a little used (e.g. its not in any active 
> project these days) previous feature of pbr/global-requirements:
> we supported things that setuptools does not: to whit, tarball and
> git requirements.
> 
> Now, these things are supported by pip, so the implementation
> involved recursing into pip from our setup.py (setup.py -> pbr ->
> pip). What we exported into setuptools was only the metadata about
> the dependency name. This meant that we were re-entering pip,
> potentially many times - it was, to be blunt, awful.
> 
> Fortunately we removed the recursive re-entry into pip in pbr 1.0. 
> This didn't remove the ability to parse requirements.txt files
> that contain urls, but it does mean they are converted to the
> simple dependency name when doing 'pip install .' in a project tree
> (or pip install $projectname), and so they are effectively
> unversioned - no lower and no upper bound. This works poorly in the
> gate: please don't use tarball or git urls in requirements.txt (or
> setup.cfg for that matter).
> 
> We can still choose to use something from git or a tarball in test 
> jobs, *if* thats the right thing (which it rarely is: I'm just
> being clear that the technical capability still exists)... but it
> needs to be done outside of requirements.txt going forward. Its
> also something that we can support with the new constraints system
> if desired [which will operate globally once in place (it is an
> extension of global-requirements)].
> 
> One question that this raises, and this is why I wrote the email:
> is there any need to support this at all:- can we say that we won't
> use tarball/vcs support at all and block it as a policy step in
> global requirements? AIUI both git and tarball support is
> problematic for CI jobs due to the increased flakiness of depending
> on network resources... so its actively harmful anyway.
> 

Lots of Neutron modules, like advanced services, or out-of-tree
plugins, rely on neutron code being checked out from git [1]. I don't
say it's the way to go forward, and there were plans to stop relying
on latest git to avoid frequent breakages, but it's not yet implemented.

[1]:
http://git.openstack.org/cgit/openstack/neutron-vpnaas/tree/tox.ini#n10

Ihar
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBCAAGBQJVcBUnAAoJEC5aWaUY1u57N2EH/jUFd0H9pQ7LApSAIlDTEl2v
WR1EXnc9Vxf5nCWq/qmncj3OCpMDlgL/ZMrFu74LRTDbe38+16kh+Fb+FvBEPGA4
ZkQC3gyg22Se/QcerTxdPil16hnT912Hr3E0cTuu/4ktyipPrVsO39N56Jbrb6WQ
SRCrEohIg7C3c0NgFcvBGh+S4rNf8IKT1oLzKrRhSLzIE8lSeGa1GNnSXPAXk19/
2KIEnqBz3Q5J6umTprB5DFdxMe93Pj6jZmGIMFaHXYgG/yTdKz3zzGM3hpuLyGUQ
kKYEzFJZ4vf2c6NBg//GYTcAkGjkM2QmAnS+uoztU5vm4QRkLgGcDCz29eQ5ufA=
=6bUu
-END PGP SIGNATURE-

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-03 Thread Thomas Goirand
On 06/03/2015 11:08 PM, Robert Collins wrote:
> Hi, right now there is a little used (e.g. its not in any active
> project these days) previous feature of pbr/global-requirements: we
> supported things that setuptools does not: to whit, tarball and git
> requirements.
> 
> Now, these things are supported by pip, so the implementation involved
> recursing into pip from our setup.py (setup.py -> pbr -> pip). What we
> exported into setuptools was only the metadata about the dependency
> name. This meant that we were re-entering pip, potentially many times
> - it was, to be blunt, awful.
> 
> Fortunately we removed the recursive re-entry into pip in pbr 1.0.
> This didn't remove the ability to parse requirements.txt files that
> contain urls, but it does mean they are converted to the simple
> dependency name when doing 'pip install .' in a project tree (or pip
> install $projectname), and so they are effectively unversioned - no
> lower and no upper bound. This works poorly in the gate: please don't
> use tarball or git urls in requirements.txt (or setup.cfg for that
> matter).
> 
> We can still choose to use something from git or a tarball in test
> jobs, *if* thats the right thing (which it rarely is: I'm just being
> clear that the technical capability still exists)... but it needs to
> be done outside of requirements.txt going forward. Its also something
> that we can support with the new constraints system if desired [which
> will operate globally once in place (it is an extension of
> global-requirements)].
> 
> One question that this raises, and this is why I wrote the email: is
> there any need to support this at all:- can we say that we won't use
> tarball/vcs support at all and block it as a policy step in global
> requirements? AIUI both git and tarball support is problematic for CI
> jobs due to the increased flakiness of depending on network
> resources... so its actively harmful anyway.
> 
> -Rob

With my package maintainer's hat on, I really would like this feature to
go away as well.

"the latest commit from this git repo" doesn't express anything
packageable, and it always feel confusing when reading it. Also, some of
my toolings (very internal automation stuff with openstack-pkg-tools for
semi-automatically maintaining Debian dependencies) don't support it also.

Thomas


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [global-requirements][pbr] tarball and git requirements no longer supported in requirements.txt

2015-06-03 Thread Robert Collins
Hi, right now there is a little used (e.g. its not in any active
project these days) previous feature of pbr/global-requirements: we
supported things that setuptools does not: to whit, tarball and git
requirements.

Now, these things are supported by pip, so the implementation involved
recursing into pip from our setup.py (setup.py -> pbr -> pip). What we
exported into setuptools was only the metadata about the dependency
name. This meant that we were re-entering pip, potentially many times
- it was, to be blunt, awful.

Fortunately we removed the recursive re-entry into pip in pbr 1.0.
This didn't remove the ability to parse requirements.txt files that
contain urls, but it does mean they are converted to the simple
dependency name when doing 'pip install .' in a project tree (or pip
install $projectname), and so they are effectively unversioned - no
lower and no upper bound. This works poorly in the gate: please don't
use tarball or git urls in requirements.txt (or setup.cfg for that
matter).

We can still choose to use something from git or a tarball in test
jobs, *if* thats the right thing (which it rarely is: I'm just being
clear that the technical capability still exists)... but it needs to
be done outside of requirements.txt going forward. Its also something
that we can support with the new constraints system if desired [which
will operate globally once in place (it is an extension of
global-requirements)].

One question that this raises, and this is why I wrote the email: is
there any need to support this at all:- can we say that we won't use
tarball/vcs support at all and block it as a policy step in global
requirements? AIUI both git and tarball support is problematic for CI
jobs due to the increased flakiness of depending on network
resources... so its actively harmful anyway.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev