Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Morgan Fainberg]

As point, we are trying to move away from this model. Having to "know" the 
dependencies is a bad experience in general. But with the move to eliminate 
"optional" parts of the api, most of these become real dependencies for 
keystone (a few things will still be optional eg memcache lib). 

--Morgan 

Sent via mobile

On Jan 30, 2015, at 17:08, Alan Pevec  wrote:

>> - Remove this requirement, no optional entries in requirements.txt, a
>> 'deployer' has to know what dependencies the components he wants to use have
> 
> Keystone is documenting its optional dependencies in test-requirements.txt
> look for # Optional ... comments in
> http://git.openstack.org/cgit/openstack/keystone/tree/test-requirements.txt
> 
> 
> Cheers,
> Alan
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Alan Pevec]

> - Remove this requirement, no optional entries in requirements.txt, a
> 'deployer' has to know what dependencies the components he wants to use have

Keystone is documenting its optional dependencies in test-requirements.txt
look for # Optional ... comments in
http://git.openstack.org/cgit/openstack/keystone/tree/test-requirements.txt


Cheers,
Alan

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Silvan Kaiser]

Hello again,
i submitted a new patch set for this at
https://review.openstack.org/#/c/110722/ ,
looking forward to further reviews. :)
Best regards
Silvan


2015-01-28 10:19 GMT+01:00 Silvan Kaiser :

> Hi All!
> Thanks for the feedback!
>
> I'll remove xattr from the requirements in my change set.
> Currently i'm working on a workaround to execute 'getfattr' instead of the
> xattr api call. We can asure getfattr is available via package dependencies
> of our client who has to be installed either way.
>
> I'm also checking out your proposal in parallel, i cannot find any
> documentation about the 'configuration management manifests', do you mean
> the puppet manifests? Otherwise, could somebody please give me a pointer to
> their documentation, etc.?
>
> Best regards
> Silvan
>
>
> 2015-01-27 18:32 GMT+01:00 Jay Pipes :
>
>> On 01/27/2015 09:13 AM, Silvan Kaiser wrote:
>>
>>> Am 27.01.2015 um 16:51 schrieb Jay Pipes :
 b) The Glance API image cache can use xattr if SQLite is not
 desired [1], and Glance does *not* list xattr as a dependency in
 requirements.txt. Swift also has a dependency on python-xattr [2].
 So, this particular Python library is not an unknown by any means.

>>> Do you happen to know how Glance handles this if the dep. is not
>>> handled in requirements.txt?
>>>
>>
>> Yep, it's considered a documentation thing and handled in configuration
>> management manifests...
>>
>> Best,
>> -jay
>>
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:
>> unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>

-- 

--
*Quobyte* GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Silvan Kaiser]

Hi All!
Thanks for the feedback!

I'll remove xattr from the requirements in my change set.
Currently i'm working on a workaround to execute 'getfattr' instead of the
xattr api call. We can asure getfattr is available via package dependencies
of our client who has to be installed either way.

I'm also checking out your proposal in parallel, i cannot find any
documentation about the 'configuration management manifests', do you mean
the puppet manifests? Otherwise, could somebody please give me a pointer to
their documentation, etc.?

Best regards
Silvan


2015-01-27 18:32 GMT+01:00 Jay Pipes :

> On 01/27/2015 09:13 AM, Silvan Kaiser wrote:
>
>> Am 27.01.2015 um 16:51 schrieb Jay Pipes :
>>> b) The Glance API image cache can use xattr if SQLite is not
>>> desired [1], and Glance does *not* list xattr as a dependency in
>>> requirements.txt. Swift also has a dependency on python-xattr [2].
>>> So, this particular Python library is not an unknown by any means.
>>>
>> Do you happen to know how Glance handles this if the dep. is not
>> handled in requirements.txt?
>>
>
> Yep, it's considered a documentation thing and handled in configuration
> management manifests...
>
> Best,
> -jay
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

-- 

--
*Quobyte* GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Sean Dague]

On 01/27/2015 08:14 AM, Daniel P. Berrange wrote:
> On Tue, Jan 27, 2015 at 07:12:29AM -0800, Sean Dague wrote:
>> On 01/27/2015 12:18 AM, Silvan Kaiser wrote:
>>> Hello!
>>> Do dependencies required only in some contexts belong into requirements.txt?
>>>
>>> Yesterday we had a short discussion on #openstack-nova regarding how to
>>> handle optional requirements. This was triggered by our quobyte nova
>>> driver (https://review.openstack.org/#/c/110722/18), who requires xattr,
>>> which we therefore added to requirements.txt (as it is provided by the
>>> requirements project).
>>>
>>> Points from the discussion:
>>> - If we add this we will be adding every requirement for every component
>>> ---> this becomes to big.
>>> - Remove this requirement, no optional entries in requirements.txt, a
>>> 'deployer' has to know what dependencies the components he wants to use have
>>> ---> Usually he does not know and installation becomes more issue prone
>>> - Other (in between) ideas???
>>>
>>> Please note that this has some urgency, the change set referenced above
>>> has been in review for months and i'm trying to react asap on comments
>>> but the deadline is approaching (next week) and if i have to do bigger
>>> changes I'd like to know as fast as possible...
>>
>> Typically the answer is no. The libvirt volume driver architecture is
>> kind of weird in the fact that it doesn't actually split it's drivers
>> out into separate files, so there can be a file level boundry (including
>> loading) for these things.
> 
> Not immediately relevant, but in the L cycle, I'm hoping to finally
> split the volume.py up into a subdirectory of files. The nova.conf
> 'volume_drivers' config will be removed by thenm, avoiding the back
> compat class naming problem we'd have if we split it in Kilo

Woot! awesome.

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Jay Pipes]

On 01/27/2015 09:13 AM, Silvan Kaiser wrote:

Am 27.01.2015 um 16:51 schrieb Jay Pipes :
b) The Glance API image cache can use xattr if SQLite is not
desired [1], and Glance does *not* list xattr as a dependency in
requirements.txt. Swift also has a dependency on python-xattr [2].
So, this particular Python library is not an unknown by any means.

Do you happen to know how Glance handles this if the dep. is not
handled in requirements.txt?


Yep, it's considered a documentation thing and handled in configuration 
management manifests...


Best,
-jay

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Silvan Kaiser]

> Am 27.01.2015 um 16:51 schrieb Jay Pipes :
> 

[…snip...]

> a) I agree with Sean and Matt here that this is an optional dependency and 
> belongs in the deployment documentation and configuration management manifests
> 
> b) The Glance API image cache can use xattr if SQLite is not desired [1], and 
> Glance does *not* list xattr as a dependency in requirements.txt. Swift also 
> has a dependency on python-xattr [2]. So, this particular Python library is 
> not an unknown by any means.
Do you happen to know how Glance handles this if the dep. is not handled in 
requirements.txt?

> 
> c) Remember that even if you install python-xattr, that still doesn't mean it 
> will automatically work. You still need to enable a filesystem that supports 
> atime (i.e. noatime must not be set in fstab for the filesystem) [3]. Just an 
> FYI.
In our case the Quobyte file system does support that metadata and we use it to 
verify that a QB volume was mounted.

Best regards
Silvan


> 
> Best,
> -jay
> 
> [1] 
> https://github.com/openstack/glance/blob/master/glance/image_cache/drivers/xattr.py
> [2] https://github.com/openstack/swift/blob/master/requirements.txt#L11
> [3] 
> https://github.com/openstack/glance/blob/master/glance/image_cache/drivers/xattr.py#L23-L24
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 

--
*Quobyte* GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Daniel P. Berrange]

On Tue, Jan 27, 2015 at 07:12:29AM -0800, Sean Dague wrote:
> On 01/27/2015 12:18 AM, Silvan Kaiser wrote:
> > Hello!
> > Do dependencies required only in some contexts belong into requirements.txt?
> > 
> > Yesterday we had a short discussion on #openstack-nova regarding how to
> > handle optional requirements. This was triggered by our quobyte nova
> > driver (https://review.openstack.org/#/c/110722/18), who requires xattr,
> > which we therefore added to requirements.txt (as it is provided by the
> > requirements project).
> > 
> > Points from the discussion:
> > - If we add this we will be adding every requirement for every component
> > ---> this becomes to big.
> > - Remove this requirement, no optional entries in requirements.txt, a
> > 'deployer' has to know what dependencies the components he wants to use have
> > ---> Usually he does not know and installation becomes more issue prone
> > - Other (in between) ideas???
> > 
> > Please note that this has some urgency, the change set referenced above
> > has been in review for months and i'm trying to react asap on comments
> > but the deadline is approaching (next week) and if i have to do bigger
> > changes I'd like to know as fast as possible...
> 
> Typically the answer is no. The libvirt volume driver architecture is
> kind of weird in the fact that it doesn't actually split it's drivers
> out into separate files, so there can be a file level boundry (including
> loading) for these things.

Not immediately relevant, but in the L cycle, I'm hoping to finally
split the volume.py up into a subdirectory of files. The nova.conf
'volume_drivers' config will be removed by thenm, avoiding the back
compat class naming problem we'd have if we split it in Kilo

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Ben Nemec]

On 01/27/2015 02:18 AM, Silvan Kaiser wrote:
> Hello!
> Do dependencies required only in some contexts belong into requirements.txt?
> 
> Yesterday we had a short discussion on #openstack-nova regarding how to
> handle optional requirements. This was triggered by our quobyte nova driver
> (https://review.openstack.org/#/c/110722/18), who requires xattr, which we
> therefore added to requirements.txt (as it is provided by the requirements
> project).
> 
> Points from the discussion:
> - If we add this we will be adding every requirement for every component
> ---> this becomes to big.
> - Remove this requirement, no optional entries in requirements.txt, a
> 'deployer' has to know what dependencies the components he wants to use have
> ---> Usually he does not know and installation becomes more issue prone
> - Other (in between) ideas???
> 
> Please note that this has some urgency, the change set referenced above has
> been in review for months and i'm trying to react asap on comments but the
> deadline is approaching (next week) and if i have to do bigger changes I'd
> like to know as fast as possible...
> 
> Best regards
> SIlvan Kaiser

I will just put in another plug for some work I had started around this
(and never finished :-( ).

https://review.openstack.org/#/c/83150/ (despite the Jenkins results,
I'm pretty sure that was working locally for me)

Also the discussion at
http://lists.openstack.org/pipermail/openstack-dev/2014-February/026976.html

There was general consensus on the approach and support has been in pbr
for a long time now, but I've never had time to figure out what would be
needed to make it play nicely with global requirements.  I _think_
that's the only major blocker preventing this from being useful, so if
someone wanted to pick it up and run with it that would be awesome.

-Ben

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Jay Pipes]

On 01/27/2015 07:14 AM, Matt Riedemann wrote:

On 1/27/2015 2:18 AM, Silvan Kaiser wrote:

Hello!
Do dependencies required only in some contexts belong into
requirements.txt?

Yesterday we had a short discussion on #openstack-nova regarding how to
handle optional requirements. This was triggered by our quobyte nova
driver (https://review.openstack.org/#/c/110722/18), who requires xattr,
which we therefore added to requirements.txt (as it is provided by the
requirements project).

Points from the discussion:
- If we add this we will be adding every requirement for every component
---> this becomes to big.
- Remove this requirement, no optional entries in requirements.txt, a
'deployer' has to know what dependencies the components he wants to
use have
---> Usually he does not know and installation becomes more issue prone
- Other (in between) ideas???

Please note that this has some urgency, the change set referenced above
has been in review for months and i'm trying to react asap on comments
but the deadline is approaching (next week) and if i have to do bigger
changes I'd like to know as fast as possible...

Best regards
SIlvan Kaiser


--
*Quobyte* GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com 
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender


__

OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



In my opinion the volume driver is optional and therefore the dependency
is optional, it's all based on what the configuration is, the same as
which DB or RPC backend you use, which is why those dependencies are in
test-requirements.txt.

While it's more obvious to a deployer that if you're going to configure
Nova to use MySQL you need some MySQL packages to make it work, any
deployer that's adding support for this volume driver should also
probably be testing their deployment scripts, e.g. chef
cookbooks/recipes, and if they haven't written their script correctly
they'll find out that it blows up with an ImportError because of a
missing xattr.  Otherwise, [1].


Couple things...

a) I agree with Sean and Matt here that this is an optional dependency 
and belongs in the deployment documentation and configuration management 
manifests.


b) The Glance API image cache can use xattr if SQLite is not desired 
[1], and Glance does *not* list xattr as a dependency in 
requirements.txt. Swift also has a dependency on python-xattr [2]. So, 
this particular Python library is not an unknown by any means.


c) Remember that even if you install python-xattr, that still doesn't 
mean it will automatically work. You still need to enable a filesystem 
that supports atime (i.e. noatime must not be set in fstab for the 
filesystem) [3]. Just an FYI.


Best,
-jay

[1] 
https://github.com/openstack/glance/blob/master/glance/image_cache/drivers/xattr.py

[2] https://github.com/openstack/swift/blob/master/requirements.txt#L11
[3] 
https://github.com/openstack/glance/blob/master/glance/image_cache/drivers/xattr.py#L23-L24


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Matt Riedemann]

On 1/27/2015 2:18 AM, Silvan Kaiser wrote:

Hello!
Do dependencies required only in some contexts belong into requirements.txt?

Yesterday we had a short discussion on #openstack-nova regarding how to
handle optional requirements. This was triggered by our quobyte nova
driver (https://review.openstack.org/#/c/110722/18), who requires xattr,
which we therefore added to requirements.txt (as it is provided by the
requirements project).

Points from the discussion:
- If we add this we will be adding every requirement for every component
---> this becomes to big.
- Remove this requirement, no optional entries in requirements.txt, a
'deployer' has to know what dependencies the components he wants to use have
---> Usually he does not know and installation becomes more issue prone
- Other (in between) ideas???

Please note that this has some urgency, the change set referenced above
has been in review for months and i'm trying to react asap on comments
but the deadline is approaching (next week) and if i have to do bigger
changes I'd like to know as fast as possible...

Best regards
SIlvan Kaiser


--
*Quobyte* GmbH
Boyenstr. 41 - 10115 Berlin-Mitte - Germany
+49-30-814 591 800 - www.quobyte.com 
Amtsgericht Berlin-Charlottenburg, HRB 149012B
management board: Dr. Felix Hupfeld, Dr. Björn Kolbeck, Dr. Jan Stender


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



In my opinion the volume driver is optional and therefore the dependency 
is optional, it's all based on what the configuration is, the same as 
which DB or RPC backend you use, which is why those dependencies are in 
test-requirements.txt.


While it's more obvious to a deployer that if you're going to configure 
Nova to use MySQL you need some MySQL packages to make it work, any 
deployer that's adding support for this volume driver should also 
probably be testing their deployment scripts, e.g. chef 
cookbooks/recipes, and if they haven't written their script correctly 
they'll find out that it blows up with an ImportError because of a 
missing xattr.  Otherwise, [1].


[1] 
http://troll.me/images/the-most-interesting-man-in-the-world/i-dont-always-test-my-code-but-when-i-do-i-do-it-in-production.jpg


--

Thanks,

Matt Riedemann


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Nova] Requirements.txt and optional requirements

[Sean Dague]

On 01/27/2015 12:18 AM, Silvan Kaiser wrote:
> Hello!
> Do dependencies required only in some contexts belong into requirements.txt?
> 
> Yesterday we had a short discussion on #openstack-nova regarding how to
> handle optional requirements. This was triggered by our quobyte nova
> driver (https://review.openstack.org/#/c/110722/18), who requires xattr,
> which we therefore added to requirements.txt (as it is provided by the
> requirements project).
> 
> Points from the discussion:
> - If we add this we will be adding every requirement for every component
> ---> this becomes to big.
> - Remove this requirement, no optional entries in requirements.txt, a
> 'deployer' has to know what dependencies the components he wants to use have
> ---> Usually he does not know and installation becomes more issue prone
> - Other (in between) ideas???
> 
> Please note that this has some urgency, the change set referenced above
> has been in review for months and i'm trying to react asap on comments
> but the deadline is approaching (next week) and if i have to do bigger
> changes I'd like to know as fast as possible...

Typically the answer is no. The libvirt volume driver architecture is
kind of weird in the fact that it doesn't actually split it's drivers
out into separate files, so there can be a file level boundry (including
loading) for these things.

That being said, in general, optional things are not in
requirements.txt. You will notice libvirt isn't even in
requirements.txt, because it's not a nova requirement.

Optional things should be handled in documentation at this point, and
the code should be structured to not fail when that's not installed if
it's not needed.

But, also, in staring at the code, I'm confused that xattr is used in
only 1 place, which is a fall back failure path. Seems like the code
could easily be redone to not need it, and get info from mount instead, no?

-Sean

-- 
Sean Dague
http://dague.net



signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev