Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Tony Breeds]

On Fri, Nov 11, 2016 at 12:08:04PM +1300, Robert Collins wrote:
> On 10 Nov 2016 9:29 PM, "Tony Breeds"  wrote:
> >
> > On Thu, Nov 10, 2016 at 08:48:16PM +1300, Robert Collins wrote:
> >
> > > We generate multiline constraints whenever versions are different
> > > between python 2 and 3 for the same package. That was happening when I
> > > wrote the tool in the first place - it was one of the reasons I wrote
> > > edit-constraints, to avoid writing silly-awkward sed. (A multiline
> > > constraint when sedd'd will error with 'requirement already supplied'
> > > or whatever the pip error string is.
> >
> > Okay.  It's unlikley that we'll hit that case but not impossible.
> 
> Currently happening to dnspython3 right now :)

Sure, but that's a little different to what we're talking about here.

I'd really like to hope[1] that we don't have any *openstack* projects that
have multiple lines of constratins.  Even if we did sed would handle that.

Regardless, we'll use edit-constratins as that'll work on windows.

Yours Tony.

[1] Yes I know "Hope is not a strategy" ;P


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Robert Collins]

On 10 Nov 2016 9:29 PM, "Tony Breeds"  wrote:
>
> On Thu, Nov 10, 2016 at 08:48:16PM +1300, Robert Collins wrote:
>
> > We generate multiline constraints whenever versions are different
> > between python 2 and 3 for the same package. That was happening when I
> > wrote the tool in the first place - it was one of the reasons I wrote
> > edit-constraints, to avoid writing silly-awkward sed. (A multiline
> > constraint when sedd'd will error with 'requirement already supplied'
> > or whatever the pip error string is.
>
> Okay.  It's unlikley that we'll hit that case but not impossible.

Currently happening to dnspython3 right now :)

> > > I think the way forward is to get openstack_requirements on pypi so
we can just
> > > pip install openstack_requirements.
> > >
> > > That'd make the script simple and still retain the
cross-platform'ness needed.
> >
> > Yes. And/or do the long mooted refactoring to separate out the scripts
> > from the current constraints and requirements.
>
> Yup, that'll be part of it but a lower priority that the actual
publication.

+1

-rob
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Tony Breeds]

On Thu, Nov 10, 2016 at 08:48:16PM +1300, Robert Collins wrote:

> We generate multiline constraints whenever versions are different
> between python 2 and 3 for the same package. That was happening when I
> wrote the tool in the first place - it was one of the reasons I wrote
> edit-constraints, to avoid writing silly-awkward sed. (A multiline
> constraint when sedd'd will error with 'requirement already supplied'
> or whatever the pip error string is.

Okay.  It's unlikley that we'll hit that case but not impossible.
 
> > I think the way forward is to get openstack_requirements on pypi so we can 
> > just
> > pip install openstack_requirements.
> >
> > That'd make the script simple and still retain the cross-platform'ness 
> > needed.
> 
> Yes. And/or do the long mooted refactoring to separate out the scripts
> from the current constraints and requirements.

Yup, that'll be part of it but a lower priority that the actual publication.

Thanks Robert

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Robert Collins]

On 10 November 2016 at 11:00, Tony Breeds  wrote:
> On Thu, Nov 10, 2016 at 06:17:44AM +1300, Robert Collins wrote:
>> Sed doesn't exist on windows, whereas a python script can. Sed doesn't
>> handle multiple line constraints.
>
> I don't think that right now we care/handle multiline constraints but the
> windows think is important and not something I considered.

We generate multiline constraints whenever versions are different
between python 2 and 3 for the same package. That was happening when I
wrote the tool in the first place - it was one of the reasons I wrote
edit-constraints, to avoid writing silly-awkward sed. (A multiline
constraint when sedd'd will error with 'requirement already supplied'
or whatever the pip error string is.

> I think the way forward is to get openstack_requirements on pypi so we can 
> just
> pip install openstack_requirements.
>
> That'd make the script simple and still retain the cross-platform'ness needed.

Yes. And/or do the long mooted refactoring to separate out the scripts
from the current constraints and requirements.

-Rob

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Tony Breeds]

On Wed, Nov 09, 2016 at 12:00:27PM +0100, Ihar Hrachyshka wrote:

> I believe sed is the way to go. There is not much we get from
> edit-constraints at this point, and it untangles the script from zuul-cloner
> that would be needed to fetch requirements repo.
> 
> It seems like the way to go. Wanna propose a patch for a repo
> (oslo.messaging) to iterate on it in gerrit?

https://review.openstack.org/#/c/394721/

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Tony Breeds]

On Thu, Nov 10, 2016 at 06:17:44AM +1300, Robert Collins wrote:
> Sed doesn't exist on windows, whereas a python script can. Sed doesn't
> handle multiple line constraints.

I don't think that right now we care/handle multiline constraints but the
windows think is important and not something I considered.

I think the way forward is to get openstack_requirements on pypi so we can just
pip install openstack_requirements.

That'd make the script simple and still retain the cross-platform'ness needed.

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Robert Collins]

Sed doesn't exist on windows, whereas a python script can. Sed doesn't
handle multiple line constraints.

Rob

On 10 Nov 2016 12:00 AM, "Ihar Hrachyshka"  wrote:

>
> > On 8 Nov 2016, at 04:07, Tony Breeds  wrote:
> >
> > On Sat, Sep 03, 2016 at 07:33:42PM +0200, Andreas Jaeger wrote:
> >> Reviewing all the constraints work, I see that repositories have created
> >> some workaround around requirements install for one of these two legimit
> >> reasons - most often using tools/tox_install.sh from tox.ini for it:
> >>
> >> 1) The repository is a dependency of another one and uses constraints,
> >> so edit upper-constraints file and allow git install.
> >>
> >> Example:
> >> http://git.openstack.org/cgit/openstack/ironic-lib/tree/
> tools/tox_install.sh
> >
> > We had a very brief discussion about this in Barcelona.  The idea being
> to
> > create an "incubator" style script in openstack/requirements that can be
> the
> > canonical source and easily copied out to repos that need it.  I'm not
> > proposing auto syncing but it would be pretty easy to script.
> >
> > We need "all projects"[1] to support constraints real soon now.  It
> seems like
> > the majority of projects that currently do not use constraints are
> because
> > they're also listed in constraints.
> >
> > I started looking at this today using [2] as the base in the
> oslo.messaging
> > repo  The good thing about this is it doesn't "just work"  I hit the
> following
> > problem:
> > ---
> > cmdargs: ['/home/stack/projects/openstack/openstack/oslo.
> messaging/tools/tox_install.sh', 'https://git.openstack.org/
> cgit/openstack/requirements/plain/upper-constraints.txt',
> '/home/stack/projects/openstack/openstack/oslo.messaging/.tox/dist/oslo.
> messaging-5.12.1.dev10.zip']
> > 
> > Processing ./.tox/dist/oslo.messaging-5.12.1.dev10.zip
> > Could not satisfy constraints for 'oslo.messaging': installation from
> path or url cannot be constrained to a version
> > ---
> >
> > This is because we use 'edit-constraints' to change
> "oslo.messaging===5.12.0" to
> > "-e file:///home/stack/projects/openstack/openstack/oslo.
> messaging#egg=oslo.messaging"[3]
> >
> > Which doesn't match because we're installing from an sdist.
> >
> > For development installs like this what we really want is a way to say
> > constrain all my requirements but allow this library to be unconstrained
> don't
> > we?  That seems to me what we're saying in [3].  When I'm working
> locally I do
> > something like:
> >
> > ---
> > pip install -c https://git.openstack.org/cgit/openstack/requirements/
> plain/upper-constraints.txt \
> >-r requirements.txt -r test-requirements.txt
> > pip install .
> > ---
> >
> > This is all leading me to think that we should just remove the
> constraint on
> > $library which can be done with something like:
> > --- [4]
> > #!/usr/bin/env bash
> >
> > # Client constraint file contains this client version pin that is in
> conflict
> > # with installing the client from source. We should remove the version
> pin in
> > # the constraints file before applying it for from-source installation.
> > BRANCH_NAME=XXX
> > CLIENT_NAME=XXX
> >
> > set -e
> >
> > CONSTRAINTS_FILE=$1
> > shift
> >
> > localfile="${VIRTUAL_ENV}/upper-constraints.txt"
> > if [[ $CONSTRAINTS_FILE != http* ]]; then
> >CONSTRAINTS_FILE=file://$CONSTRAINTS_FILE
> > fi
> >
> > curl $CONSTRAINTS_FILE -k -o $localfile
> > sed -i~ -e "/^${CLIENT_NAME}===/d" $localfile
> >
> > pip install -U -c$localfile $*
> > ---
> >
> > Using openstack_requirements is the "right" thing to do and we could
> still use
> > edit-constraints $localfile -- $CLIENT_NAME ""
> > do delete the entry but it seems like wasted work to me
>
> I believe sed is the way to go. There is not much we get from
> edit-constraints at this point, and it untangles the script from
> zuul-cloner that would be needed to fetch requirements repo.
>
> It seems like the way to go. Wanna propose a patch for a repo
> (oslo.messaging) to iterate on it in gerrit?
>
> Ihar
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Ihar Hrachyshka]

> On 8 Nov 2016, at 04:07, Tony Breeds  wrote:
> 
> On Sat, Sep 03, 2016 at 07:33:42PM +0200, Andreas Jaeger wrote:
>> Reviewing all the constraints work, I see that repositories have created
>> some workaround around requirements install for one of these two legimit
>> reasons - most often using tools/tox_install.sh from tox.ini for it:
>> 
>> 1) The repository is a dependency of another one and uses constraints,
>> so edit upper-constraints file and allow git install.
>> 
>> Example:
>> http://git.openstack.org/cgit/openstack/ironic-lib/tree/tools/tox_install.sh
> 
> We had a very brief discussion about this in Barcelona.  The idea being to
> create an "incubator" style script in openstack/requirements that can be the
> canonical source and easily copied out to repos that need it.  I'm not
> proposing auto syncing but it would be pretty easy to script.
> 
> We need "all projects"[1] to support constraints real soon now.  It seems like
> the majority of projects that currently do not use constraints are because
> they're also listed in constraints.
> 
> I started looking at this today using [2] as the base in the oslo.messaging
> repo  The good thing about this is it doesn't "just work"  I hit the following
> problem:
> ---
> cmdargs: 
> ['/home/stack/projects/openstack/openstack/oslo.messaging/tools/tox_install.sh',
>  
> 'https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt',
>  
> '/home/stack/projects/openstack/openstack/oslo.messaging/.tox/dist/oslo.messaging-5.12.1.dev10.zip']
> 
> Processing ./.tox/dist/oslo.messaging-5.12.1.dev10.zip
> Could not satisfy constraints for 'oslo.messaging': installation from path or 
> url cannot be constrained to a version
> ---
> 
> This is because we use 'edit-constraints' to change "oslo.messaging===5.12.0" 
> to
> "-e 
> file:///home/stack/projects/openstack/openstack/oslo.messaging#egg=oslo.messaging"[3]
> 
> Which doesn't match because we're installing from an sdist.
> 
> For development installs like this what we really want is a way to say
> constrain all my requirements but allow this library to be unconstrained don't
> we?  That seems to me what we're saying in [3].  When I'm working locally I do
> something like:
> 
> ---
> pip install -c 
> https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt
>  \
>-r requirements.txt -r test-requirements.txt
> pip install .
> ---
> 
> This is all leading me to think that we should just remove the constraint on
> $library which can be done with something like:
> --- [4]
> #!/usr/bin/env bash
> 
> # Client constraint file contains this client version pin that is in conflict
> # with installing the client from source. We should remove the version pin in
> # the constraints file before applying it for from-source installation.
> BRANCH_NAME=XXX
> CLIENT_NAME=XXX
> 
> set -e
> 
> CONSTRAINTS_FILE=$1
> shift
> 
> localfile="${VIRTUAL_ENV}/upper-constraints.txt"
> if [[ $CONSTRAINTS_FILE != http* ]]; then
>CONSTRAINTS_FILE=file://$CONSTRAINTS_FILE
> fi
> 
> curl $CONSTRAINTS_FILE -k -o $localfile
> sed -i~ -e "/^${CLIENT_NAME}===/d" $localfile
> 
> pip install -U -c$localfile $*
> ---
> 
> Using openstack_requirements is the "right" thing to do and we could still use
> edit-constraints $localfile -- $CLIENT_NAME ""
> do delete the entry but it seems like wasted work to me

I believe sed is the way to go. There is not much we get from edit-constraints 
at this point, and it untangles the script from zuul-cloner that would be 
needed to fetch requirements repo.

It seems like the way to go. Wanna propose a patch for a repo (oslo.messaging) 
to iterate on it in gerrit?

Ihar
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Tony Breeds]

On Sat, Sep 03, 2016 at 07:33:42PM +0200, Andreas Jaeger wrote:
> Reviewing all the constraints work, I see that repositories have created
> some workaround around requirements install for one of these two legimit
> reasons - most often using tools/tox_install.sh from tox.ini for it:
> 
> 1) The repository is a dependency of another one and uses constraints,
> so edit upper-constraints file and allow git install.
> 
> Example:
> http://git.openstack.org/cgit/openstack/ironic-lib/tree/tools/tox_install.sh

We had a very brief discussion about this in Barcelona.  The idea being to
create an "incubator" style script in openstack/requirements that can be the
canonical source and easily copied out to repos that need it.  I'm not
proposing auto syncing but it would be pretty easy to script.

We need "all projects"[1] to support constraints real soon now.  It seems like
the majority of projects that currently do not use constraints are because
they're also listed in constraints.

I started looking at this today using [2] as the base in the oslo.messaging
repo  The good thing about this is it doesn't "just work"  I hit the following
problem:
---
cmdargs: 
['/home/stack/projects/openstack/openstack/oslo.messaging/tools/tox_install.sh',
 
'https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt',
 
'/home/stack/projects/openstack/openstack/oslo.messaging/.tox/dist/oslo.messaging-5.12.1.dev10.zip']

Processing ./.tox/dist/oslo.messaging-5.12.1.dev10.zip
Could not satisfy constraints for 'oslo.messaging': installation from path or 
url cannot be constrained to a version
---

This is because we use 'edit-constraints' to change "oslo.messaging===5.12.0" to
"-e 
file:///home/stack/projects/openstack/openstack/oslo.messaging#egg=oslo.messaging"[3]

Which doesn't match because we're installing from an sdist.

For development installs like this what we really want is a way to say
constrain all my requirements but allow this library to be unconstrained don't
we?  That seems to me what we're saying in [3].  When I'm working locally I do
something like:

---
pip install -c 
https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt
 \
-r requirements.txt -r test-requirements.txt
pip install .
---

This is all leading me to think that we should just remove the constraint on
$library which can be done with something like:
--- [4]
#!/usr/bin/env bash

# Client constraint file contains this client version pin that is in conflict
# with installing the client from source. We should remove the version pin in
# the constraints file before applying it for from-source installation.
BRANCH_NAME=XXX
CLIENT_NAME=XXX

set -e

CONSTRAINTS_FILE=$1
shift

localfile="${VIRTUAL_ENV}/upper-constraints.txt"
if [[ $CONSTRAINTS_FILE != http* ]]; then
CONSTRAINTS_FILE=file://$CONSTRAINTS_FILE
fi

curl $CONSTRAINTS_FILE -k -o $localfile
sed -i~ -e "/^${CLIENT_NAME}===/d" $localfile

pip install -U -c$localfile $*
---

Using openstack_requirements is the "right" thing to do and we could still use
edit-constraints $localfile -- $CLIENT_NAME ""
do delete the entry but it seems like wasted work to me

> 2) Install a package that is not on pypi. This is typical neutron or
> horizon.
> 
> Example:
> http://git.openstack.org/cgit/openstack/neutron-lbaas/tree/tools/tox_install.sh

This is much less thought out but can't we enhance the deps in tox.ini like:

---
deps = os:neutron
   os:neutron-lbaas
   -r requirements.txt
---

and then use something like:
--- [5]
#!/usr/bin/env bash

function install_os_dep()
{
   set -- $( echo $1 | sed -e 's/:/ /')
   prefix=$1  # ignored
   project=$2
   
   # Now basically turn 
http://git.openstack.org/cgit/openstack/neutron-lbaas/tree/tools/tox_install.sh 
into a shell function
}
# Client constraint file contains this client version pin that is in conflict
# with installing the client from source. We should remove the version pin in
# the constraints file before applying it for from-source installation.
BRANCH_NAME=XXX
CLIENT_NAME=XXX

set -e

CONSTRAINTS_FILE=$1
shift

localfile="${VIRTUAL_ENV}/upper-constraints.txt"
if [[ $CONSTRAINTS_FILE != http* ]]; then
CONSTRAINTS_FILE=file://$CONSTRAINTS_FILE
fi

curl $CONSTRAINTS_FILE -k -o $localfile
sed -i~ -e "/^${CLIENT_NAME}===/d" $localfile

deps=""
while [ $# -gt 1 ] ; do
case "$1" in
os:*)
install_os_dep "$1"
;;
*)
  deps+="$1"
;;
shift 1
done
pip install -U -c$localfile $deps
exit $?
--- [6]

We also discussed that the right place to fix these issues could be in tox
and/or pip but as I said earlier it's a matter of balancing right vs right now
:(

Yours Tony.

[1] For now that means all deliverables listed in
openstack/governance:reference/projects.yaml, but nothing is stopping
adoption in other stackforge projects.
[2] 
http://git.openstack.org/cgit/openstack/python-neutronclient/tree/tools/tox_install.sh
[3] 

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Ihar Hrachyshka]

Andreas Jaeger  wrote:


On 09/05/2016 07:45 AM, Tony Breeds wrote:

On Sat, Sep 03, 2016 at 07:33:42PM +0200, Andreas Jaeger wrote:

Reviewing all the constraints work, I see that repositories have created
some workaround around requirements install for one of these two legimit
reasons - most often using tools/tox_install.sh from tox.ini for it:

1) The repository is a dependency of another one and uses constraints,
so edit upper-constraints file and allow git install.

Example:
http://git.openstack.org/cgit/openstack/ironic-lib/tree/tools/tox_install.sh

2) Install a package that is not on pypi. This is typical neutron or
horizon.

Example:
http://git.openstack.org/cgit/openstack/neutron-lbaas/tree/tools/tox_install.sh

Note there's a third usage as well: Install packages that are not in
global-requirements but let's leave that use case out of this discussion.

For 2, Robert Collins has started in
https://review.openstack.org/#/c/252680 a way to check out the required
repos - but that change is incomplete and looks orphaned.

Instead of several projects copying and adopting these files - and thus
having them in various state in repositories - what can we do to make it
easier for developer and projects to address these two problems?


Hi Andreas,
Thanks for bringing this up, it's been on my radar for a while but I haven't
made much forward progress apart from a few quick conversations.

Fixing the first item you point out need to be done in pip, and as  
always there
is a balance between doing it fast and doing it right.  The requirements  
team
has this item on it's todo list for Ocata.  In the meantime we could  
look at

simplifying the scripts in use if that's of any benefit.


Glad to hear!

The second item has always confused me slightly.  I'm not sure why  
projects

that need horizon/neutron have a tox_install script, can't they just add
something like:

"-e git://git.openstack.org/openstack/neutron@master#egg=neutron"
or
"http://tarballs.openstack.org/horizon/horizon-master.tar.gz#egg=horizon;

in test-requirements.txt ?  Actually I see from [1] that quite a few  
horizon

plugins are doing just that.  I wonder if that'd work better for neutron?

Yours Tony.

[1]  
http://codesearch.openstack.org/?q=horizon-master.tar.gz=nope=test-requirements.txt=


Indeed, these work in general - but not with Depends-On, so if you want
to test a proposed change in horizon together with a change in say
mistral-ui.

Not sure how much that is a problem for horizon or neutron,


We use Depends-On quite actively to coordinate proper order of landing  
patches in 5 repos we currently maintain as neutron core, plus a vast  
number of drivers/plugins we have in stadium and outside of it. Not having  
the tool would be a bummer for the way neutron team does development.


Ihar

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Akihiro Motoki]

2016-09-05 16:03 GMT+09:00 Andreas Jaeger :
> On 09/05/2016 07:45 AM, Tony Breeds wrote:
>> On Sat, Sep 03, 2016 at 07:33:42PM +0200, Andreas Jaeger wrote:
>>> Reviewing all the constraints work, I see that repositories have created
>>> some workaround around requirements install for one of these two legimit
>>> reasons - most often using tools/tox_install.sh from tox.ini for it:
>>>
>>> 1) The repository is a dependency of another one and uses constraints,
>>> so edit upper-constraints file and allow git install.
>>>
>>> Example:
>>> http://git.openstack.org/cgit/openstack/ironic-lib/tree/tools/tox_install.sh
>>>
>>> 2) Install a package that is not on pypi. This is typical neutron or
>>> horizon.
>>>
>>> Example:
>>> http://git.openstack.org/cgit/openstack/neutron-lbaas/tree/tools/tox_install.sh
>>>
>>> Note there's a third usage as well: Install packages that are not in
>>> global-requirements but let's leave that use case out of this discussion.
>>>
>>> For 2, Robert Collins has started in
>>> https://review.openstack.org/#/c/252680 a way to check out the required
>>> repos - but that change is incomplete and looks orphaned.
>>>
>>> Instead of several projects copying and adopting these files - and thus
>>> having them in various state in repositories - what can we do to make it
>>> easier for developer and projects to address these two problems?
>>
>> Hi Andreas,
>> Thanks for bringing this up, it's been on my radar for a while but I 
>> haven't
>> made much forward progress apart from a few quick conversations.
>>
>> Fixing the first item you point out need to be done in pip, and as always 
>> there
>> is a balance between doing it fast and doing it right.  The requirements team
>> has this item on it's todo list for Ocata.  In the meantime we could look at
>> simplifying the scripts in use if that's of any benefit.
>
> Glad to hear!
>
>> The second item has always confused me slightly.  I'm not sure why projects
>> that need horizon/neutron have a tox_install script, can't they just add
>> something like:
>>
>> "-e git://git.openstack.org/openstack/neutron@master#egg=neutron"
>> or
>> "http://tarballs.openstack.org/horizon/horizon-master.tar.gz#egg=horizon;
>>
>> in test-requirements.txt ?  Actually I see from [1] that quite a few horizon
>> plugins are doing just that.  I wonder if that'd work better for neutron?
>>
>> Yours Tony.
>>
>> [1] 
>> http://codesearch.openstack.org/?q=horizon-master.tar.gz=nope=test-requirements.txt=
>
> Indeed, these work in general - but not with Depends-On, so if you want
> to test a proposed change in horizon together with a change in say
> mistral-ui.
>
> Not sure how much that is a problem for horizon or neutron,

IIRC, neutron introduced tox_install.sh first and
I think it was to support Depends-On mechanism.
The main histories are found in openstack/neutron-lbaas repo:
c61f920f2a54fd829ee37484b3743ab77b909847 (tox_install.sh was introduced)
eb41ab78b87f1589fada5fca0c9189fa361d6289 (stop specifying neutron in
requirements.txt)

For Horizon, I am the person who started to tox_install.sh.
Horizon plugins did not use Depends-On to Horiozn patch frequently.
The main reason was to fix https://bugs.launchpad.net/devstack/+bug/1540328 .
It is similar situation to neutron.

Hope this helps you.

Thanks,
Akihiro


>
> Andreas
> --
>  Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
>   SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
>GF: Felix Imendörffer, Jane Smithard, Graham Norton,
>HRB 21284 (AG Nürnberg)
> GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Andreas Jaeger]

On 09/05/2016 07:45 AM, Tony Breeds wrote:
> On Sat, Sep 03, 2016 at 07:33:42PM +0200, Andreas Jaeger wrote:
>> Reviewing all the constraints work, I see that repositories have created
>> some workaround around requirements install for one of these two legimit
>> reasons - most often using tools/tox_install.sh from tox.ini for it:
>>
>> 1) The repository is a dependency of another one and uses constraints,
>> so edit upper-constraints file and allow git install.
>>
>> Example:
>> http://git.openstack.org/cgit/openstack/ironic-lib/tree/tools/tox_install.sh
>>
>> 2) Install a package that is not on pypi. This is typical neutron or
>> horizon.
>>
>> Example:
>> http://git.openstack.org/cgit/openstack/neutron-lbaas/tree/tools/tox_install.sh
>>
>> Note there's a third usage as well: Install packages that are not in
>> global-requirements but let's leave that use case out of this discussion.
>>
>> For 2, Robert Collins has started in
>> https://review.openstack.org/#/c/252680 a way to check out the required
>> repos - but that change is incomplete and looks orphaned.
>>
>> Instead of several projects copying and adopting these files - and thus
>> having them in various state in repositories - what can we do to make it
>> easier for developer and projects to address these two problems?
> 
> Hi Andreas,
> Thanks for bringing this up, it's been on my radar for a while but I 
> haven't
> made much forward progress apart from a few quick conversations.
> 
> Fixing the first item you point out need to be done in pip, and as always 
> there
> is a balance between doing it fast and doing it right.  The requirements team
> has this item on it's todo list for Ocata.  In the meantime we could look at
> simplifying the scripts in use if that's of any benefit.

Glad to hear!

> The second item has always confused me slightly.  I'm not sure why projects
> that need horizon/neutron have a tox_install script, can't they just add
> something like:
> 
> "-e git://git.openstack.org/openstack/neutron@master#egg=neutron"
> or
> "http://tarballs.openstack.org/horizon/horizon-master.tar.gz#egg=horizon;
> 
> in test-requirements.txt ?  Actually I see from [1] that quite a few horizon
> plugins are doing just that.  I wonder if that'd work better for neutron?
> 
> Yours Tony.
> 
> [1] 
> http://codesearch.openstack.org/?q=horizon-master.tar.gz=nope=test-requirements.txt=

Indeed, these work in general - but not with Depends-On, so if you want
to test a proposed change in horizon together with a change in say
mistral-ui.

Not sure how much that is a problem for horizon or neutron,

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Tony Breeds]

On Sat, Sep 03, 2016 at 07:33:42PM +0200, Andreas Jaeger wrote:
> Reviewing all the constraints work, I see that repositories have created
> some workaround around requirements install for one of these two legimit
> reasons - most often using tools/tox_install.sh from tox.ini for it:
> 
> 1) The repository is a dependency of another one and uses constraints,
> so edit upper-constraints file and allow git install.
> 
> Example:
> http://git.openstack.org/cgit/openstack/ironic-lib/tree/tools/tox_install.sh
> 
> 2) Install a package that is not on pypi. This is typical neutron or
> horizon.
> 
> Example:
> http://git.openstack.org/cgit/openstack/neutron-lbaas/tree/tools/tox_install.sh
> 
> Note there's a third usage as well: Install packages that are not in
> global-requirements but let's leave that use case out of this discussion.
> 
> For 2, Robert Collins has started in
> https://review.openstack.org/#/c/252680 a way to check out the required
> repos - but that change is incomplete and looks orphaned.
> 
> Instead of several projects copying and adopting these files - and thus
> having them in various state in repositories - what can we do to make it
> easier for developer and projects to address these two problems?

Hi Andreas,
Thanks for bringing this up, it's been on my radar for a while but I haven't
made much forward progress apart from a few quick conversations.

Fixing the first item you point out need to be done in pip, and as always there
is a balance between doing it fast and doing it right.  The requirements team
has this item on it's todo list for Ocata.  In the meantime we could look at
simplifying the scripts in use if that's of any benefit.

The second item has always confused me slightly.  I'm not sure why projects
that need horizon/neutron have a tox_install script, can't they just add
something like:

"-e git://git.openstack.org/openstack/neutron@master#egg=neutron"
or
"http://tarballs.openstack.org/horizon/horizon-master.tar.gz#egg=horizon;

in test-requirements.txt ?  Actually I see from [1] that quite a few horizon
plugins are doing just that.  I wonder if that'd work better for neutron?

Yours Tony.

[1] 
http://codesearch.openstack.org/?q=horizon-master.tar.gz=nope=test-requirements.txt=


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [all][infra][requirements] Odd cases in requirements?

[Andreas Jaeger]

Reviewing all the constraints work, I see that repositories have created
some workaround around requirements install for one of these two legimit
reasons - most often using tools/tox_install.sh from tox.ini for it:

1) The repository is a dependency of another one and uses constraints,
so edit upper-constraints file and allow git install.

Example:
http://git.openstack.org/cgit/openstack/ironic-lib/tree/tools/tox_install.sh

2) Install a package that is not on pypi. This is typical neutron or
horizon.

Example:
http://git.openstack.org/cgit/openstack/neutron-lbaas/tree/tools/tox_install.sh

Note there's a third usage as well: Install packages that are not in
global-requirements but let's leave that use case out of this discussion.

For 2, Robert Collins has started in
https://review.openstack.org/#/c/252680 a way to check out the required
repos - but that change is incomplete and looks orphaned.

Instead of several projects copying and adopting these files - and thus
having them in various state in repositories - what can we do to make it
easier for developer and projects to address these two problems?

Andreas
-- 
 Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi
  SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
   GF: Felix Imendörffer, Jane Smithard, Graham Norton,
   HRB 21284 (AG Nürnberg)
GPG fingerprint = 93A3 365E CE47 B889 DF7F  FED1 389A 563C C272 A126


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev