Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

+1 for opening new threads regarding specific questions.

On 21 Jan 2014, at 11:54, Renat Akhmerov  wrote:

> 
> On 21 Jan 2014, at 09:40, Sean Dague  wrote:
> 
>> On 01/21/2014 11:54 AM, Renat Akhmerov wrote:
>>> 
>>> On 17 Jan 2014, at 22:00, Jamie Lennox >> > wrote:
>>> 
 (I don't buy the problem with large amounts of dependencies, if you
 have a meta-package you just have one line in requirements and pip
 will figure the rest out.)
>>> 
>>> +1
>>> 
>>> Renat Akhmerov
>>> @ Mirantis Inc.
>> 
>> Man, where were you then when we had to spend 3 weeks unwinding global
>> requirements in the gate because pip was figuring it out all kinds of
>> wrong, and we'd do things like uninstall and reinstall
>> python-keystoneclient 6 times during an install. Because after that
>> experience, I'm very anti "pip will figure the rest out”.
> 
> Honestly, I was very far but now I’m much closer :)
> 
>> Because it won't, not in python, where we're talking about libraries
>> that are in the global namespace, where python can only have 1 version
>> of a dependency installed.
>> 
>> If the the solution is every openstack project should install a venv for
>> all it's dependencies to get around this issue, then we're talking a
>> different problem (and a different architecture from what we've been
>> trying to do). But I find the idea of having 12 copies of
>> python-keystone client installed on my openstack environment to be
>> distasteful.
> 
> I see your point. Right now this is really a problem. I hope it’ll be solved 
> someday globally so that python could have more than 1 version of a library. 
> However, some organizational steps could be made to strictly control versions 
> of libraries.
> 
> Anyway, I admit you’re right here.
> 
>> So come spend a month working on requirements updates in OpenStack
>> gate…
> 
> Frankly, never had significant experience in that. I would love to :)
> 
>> well you are a braver man than I. :)
> 
> Nope, it’s hardly true :)

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

On 21 Jan 2014, at 09:40, Sean Dague  wrote:

> On 01/21/2014 11:54 AM, Renat Akhmerov wrote:
>> 
>> On 17 Jan 2014, at 22:00, Jamie Lennox > > wrote:
>> 
>>> (I don't buy the problem with large amounts of dependencies, if you
>>> have a meta-package you just have one line in requirements and pip
>>> will figure the rest out.)
>> 
>> +1
>> 
>> Renat Akhmerov
>> @ Mirantis Inc.
> 
> Man, where were you then when we had to spend 3 weeks unwinding global
> requirements in the gate because pip was figuring it out all kinds of
> wrong, and we'd do things like uninstall and reinstall
> python-keystoneclient 6 times during an install. Because after that
> experience, I'm very anti "pip will figure the rest out”.

Honestly, I was very far but now I’m much closer :)

> Because it won't, not in python, where we're talking about libraries
> that are in the global namespace, where python can only have 1 version
> of a dependency installed.
> 
> If the the solution is every openstack project should install a venv for
> all it's dependencies to get around this issue, then we're talking a
> different problem (and a different architecture from what we've been
> trying to do). But I find the idea of having 12 copies of
> python-keystone client installed on my openstack environment to be
> distasteful.

I see your point. Right now this is really a problem. I hope it’ll be solved 
someday globally so that python could have more than 1 version of a library. 
However, some organizational steps could be made to strictly control versions 
of libraries.

Anyway, I admit you’re right here.

> So come spend a month working on requirements updates in OpenStack
> gate…

Frankly, never had significant experience in that. I would love to :)

> well you are a braver man than I. :)

Nope, it’s hardly true :)



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

On 21 Jan 2014, at 09:07, Jesse Noller  wrote:

> Do you use any other platform than Linux? Even donald - one of the python 
> packaging leads and PyPI leads said this is a bad end-user experience for 
> consumers of openstack clouds.

That fact that someone (even very smart experience) said something doesn’t mean 
we should stop thinking forward. I am pretty new to Python though and apologize 
if I misunderstand things sometimes.___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 21, 2014, at 12:59 PM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:

>It is when most openstack clouds don’t just run keystone. Or nova, or swift. 
>Or when each client acts, smells and behaves differently. It matters a LOT 
>when you’re trying to write applications on top of a mature openstack 
>deployment.

I still don't understand the problem here. Installed packages usually just lie 
quietly on your disk (which size is now measured in terabytes) and they don't 
act or stink unless you ask them to. I'm pretty sure that most of the people 
are not aware of ALL packages installed in their distribution and few packages 
that are not used won't make it any worse


>I have actual experience here as a person running a cloud and supporting 
>end-users & developers: the overwhelming customer feedback (paid and unpaid 
>(exploratory users)) is that the 22+ clients are confusing, difficult to use, 
>prone to error. There are two ways of resolving this if you’re in my shoes - 
>or in a role where the primary focus is not openstack developers or builders; 
>the first is you coordinate work focusing on developer & end user experience 
>upstream, working with openstack and the teams to come up with something that 
>benefits everyone, or, you fork, and build the openstack equivalent to boto / 
>awscli so you can provide a unified “one obvious way to consume openstack 
>clouds”.

I agree that many clients can be confusing however I think that actuall problem 
here is not the clients number but discrepancies in client parameters naming 
and missing/unclear help.
In our work of clients refactoring I will also update bash completition that we 
have for our clients. So when you use nova start-server you will see options 
applicable to this command etc.

For the unified client we can make a nice commands hierarhcy and I don't think 
it will be confusing to users for example:

$ openstack help
nova cloud computing fabric controller
glance  discovering, registering, retrieving and storing virtual machine 
images
cinder
...
22 records

$ openstack help nova
OpenStack Nova provides a cloud computing fabric controller, supporting a wide 
variety of virtualization technologies, including KVM, Xen, LXC, VMware, and 
more. In addition to its native API, it includes compatibility with the 
commonly encountered Amazon EC2 and S3 APIs.

start-server
...

With proper bash completition it should be quite easy to use.


Since people are complaining about thread depth: I’ll focus on the blueprint 
and wiki page for now - we’re disagreeing on basic assumptions and facts here. 
If you are a developer or a user, trying to build an application that even uses 
50% of the currently shipping openstack software; your life is extremely sub 
optimal.

The ideas expressed is to replace the focus of the clients and CLIs from “those 
who build openstack” to “those that consume openstack” which is a very 
different profile, knowledge level and user story.



2014/1/21 Jesse Noller 
mailto:jesse.nol...@rackspace.com>>

On Jan 21, 2014, at 12:19 PM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:

Hello,

I would like to end this requirements talk cause it doesn't make any sense in 
term of python clients.
Initially the discussion was about "many clients projects with separate 
requirements" VS "single client project with single requirements list".


Then I suspect we’re at an impasse; I do plan on proceeding with a new wiki 
page and blueprint for a unified client, SDK (backend) for end users. I for one 
do not think things are as clear cut as you make them - and the +1s to the 
unified client thoughts clearly express the discontent with the current 
structure & packaging.

At that moment we should have stop and actually open requirements lists for 
python clients.
Basically all clients have the same requirement (cause they all do the same 
stuff - sending HTTP requests K.O.)
There is absolutely no difference in the situation of many clients vs single 
client.

Answering another question about "user only needs X (keystone) and we install 
package with clients for all openstack services":
Size of keystone client (and any other client I suppose) is ~300Kb I don't 
think that it's a big difference for the user to install package that is ~300Kb 
or ~10Mb (unless we are using openstack from Android).

It is when most openstack clouds don’t just run keystone. Or nova, or swift. Or 
when each client acts, smells and behaves differently. It matters a LOT when 
you’re trying to write applications on top of a mature openstack deployment.


>From the user perspective I think it's much easier to use client with 
>"everything included" rather than try to google for client package for some 
>rarely used service.


I have actual experience here as a person running a cloud and supporting 
end-users & developers: the overwhelming customer feedback (paid and unpaid 
(exploratory users)) is that the 22+ clients are confusin

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

>It is when most openstack clouds don’t just run keystone. Or nova, or
swift. Or when each client acts, smells and behaves differently. It matters
a LOT when you’re trying to write applications on top of a mature openstack
deployment.

I still don't understand the problem here. Installed packages usually just
lie quietly on your disk (which size is now measured in terabytes) and they
don't act or stink unless you ask them to. I'm pretty sure that most of the
people are not aware of ALL packages installed in their distribution and
few packages that are not used won't make it any worse


 >I have actual experience here as a person running a cloud and supporting
end-users & developers: the overwhelming customer feedback (paid and unpaid
(exploratory users)) is that the 22+ clients are confusing, difficult to
use, prone to error. There are two ways of resolving this if you’re in my
shoes - or in a role where the primary focus is not openstack developers or
builders; the first is you coordinate work focusing on developer & end user
experience upstream, working with openstack and the teams to come up with
something that benefits everyone, or, you fork, and build the openstack
equivalent to boto / awscli so you can provide a unified “one obvious way
to consume openstack clouds”.

I agree that many clients can be confusing however I think that actuall
problem here is not the clients number but discrepancies in client
parameters naming and missing/unclear help.
In our work of clients refactoring I will also update bash completition
that we have for our clients. So when you use nova start-server you will
see options applicable to this command etc.

For the unified client we can make a nice commands hierarhcy and I don't
think it will be confusing to users for example:

$ openstack help
nova cloud computing fabric controller
glance  discovering, registering, retrieving and storing virtual
machine images
cinder
...
22 records

$ openstack help nova
OpenStack Nova provides a cloud computing fabric controller, supporting a
wide variety of virtualization technologies, including KVM, Xen, LXC,
VMware, and more. In addition to its native API, it includes compatibility
with the commonly encountered Amazon EC2 and S3 APIs.

start-server
...

With proper bash completition it should be quite easy to use.



2014/1/21 Jesse Noller 

>
>  On Jan 21, 2014, at 12:19 PM, Alexei Kornienko <
> alexei.kornie...@gmail.com> wrote:
>
>  Hello,
>
> I would like to end this requirements talk cause it doesn't make any sense
> in term of python clients.
>  Initially the discussion was about "many clients projects with separate
> requirements" VS "single client project with single requirements list".
>
>
>  Then I suspect we’re at an impasse; I do plan on proceeding with a new
> wiki page and blueprint for a unified client, SDK (backend) for end users.
> I for one do not think things are as clear cut as you make them - and the
> +1s to the unified client thoughts clearly express the discontent with the
> current structure & packaging.
>
>  At that moment we should have stop and actually open requirements lists
> for python clients.
> Basically all clients have the same requirement (cause they all do the
> same stuff - sending HTTP requests K.O.)
>  There is absolutely no difference in the situation of many clients vs
> single client.
>
>  Answering another question about "user only needs X (keystone) and we
> install package with clients for all openstack services":
>  Size of keystone client (and any other client I suppose) is ~300Kb I
> don't think that it's a big difference for the user to install package that
> is ~300Kb or ~10Mb (unless we are using openstack from Android).
>
>
>  It is when most openstack clouds don’t just run keystone. Or nova, or
> swift. Or when each client acts, smells and behaves differently. It matters
> a LOT when you’re trying to write applications on top of a mature openstack
> deployment.
>
>
>  From the user perspective I think it's much easier to use client with
> "everything included" rather than try to google for client package for some
> rarely used service.
>
>
>  I have actual experience here as a person running a cloud and supporting
> end-users & developers: the overwhelming customer feedback (paid and unpaid
> (exploratory users)) is that the 22+ clients are confusing, difficult to
> use, prone to error. There are two ways of resolving this if you’re in my
> shoes - or in a role where the primary focus is not openstack developers or
> builders; the first is you coordinate work focusing on developer & end user
> experience upstream, working with openstack and the teams to come up with
> something that benefits everyone, or, you fork, and build the openstack
> equivalent to boto / awscli so you can provide a unified “one obvious way
> to consume openstack clouds”.
>
>  jesse
>
>  Regards,
> Alexei
>
>
>
>
> 2014/1/21 Sean Dague 
>
>> On 01/21/2014 11:54 AM, Renat Akhmerov wrote:
>> >
>> > O

Re: [openstack-dev] a "common" client library

[Jamie Lennox]

don't do this here - the thread is way too deep already. 

If we get into discussing individual points let's do one question per thread 
and prefix the emails with [client] or something to tie it all together

- Original Message -
> From: "Alexei Kornienko" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Wednesday, 22 January, 2014 4:35:39 AM
> Subject: Re: [openstack-dev] a "common" client library
> 
> I would like to propose to use this thread to gather and discuss software
> requirements that our clients should meet.
> Later we'll summarize all the requirements and use them during our work of
> improving the clients.
> By reaching listed requirements we'll be able to evaluate the success of our
> refactoring.
> 
> To start this discussion:
> * API versioning should be encapsulated inside of the client.
> 
> From the end user perspective using the code should look something like:
> 
> import openstack.nova
> 
> nova = openstack.nova.create_client(...params TBD...)
> nova.start_server(...)
> 
> Reasoning for this requirement:
> If user would have to specify version explicitly his code will only work with
> specific version of client/api or user would have to write several code
> paths for different api versions.
> 
> Implementation:
> We will have an interface (abstact class) that would contain all the methods
> available in latest API.
> Different versions of the API would implement this methods. In case method
> cannot be implemented with existing API it should raise
> "UnsupportedOperation" exception.
> 
> Glossary:
> user - developer who uses the client/openstack API
> 
> Please share your thoughts about this requirement/other requirements.
> 
> Looking forward for your replies,
> Alexei Kornienko
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 21, 2014, at 12:19 PM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:

Hello,

I would like to end this requirements talk cause it doesn't make any sense in 
term of python clients.
Initially the discussion was about "many clients projects with separate 
requirements" VS "single client project with single requirements list".


Then I suspect we’re at an impasse; I do plan on proceeding with a new wiki 
page and blueprint for a unified client, SDK (backend) for end users. I for one 
do not think things are as clear cut as you make them - and the +1s to the 
unified client thoughts clearly express the discontent with the current 
structure & packaging.

At that moment we should have stop and actually open requirements lists for 
python clients.
Basically all clients have the same requirement (cause they all do the same 
stuff - sending HTTP requests K.O.)
There is absolutely no difference in the situation of many clients vs single 
client.

Answering another question about "user only needs X (keystone) and we install 
package with clients for all openstack services":
Size of keystone client (and any other client I suppose) is ~300Kb I don't 
think that it's a big difference for the user to install package that is ~300Kb 
or ~10Mb (unless we are using openstack from Android).

It is when most openstack clouds don’t just run keystone. Or nova, or swift. Or 
when each client acts, smells and behaves differently. It matters a LOT when 
you’re trying to write applications on top of a mature openstack deployment.


>From the user perspective I think it's much easier to use client with 
>"everything included" rather than try to google for client package for some 
>rarely used service.


I have actual experience here as a person running a cloud and supporting 
end-users & developers: the overwhelming customer feedback (paid and unpaid 
(exploratory users)) is that the 22+ clients are confusing, difficult to use, 
prone to error. There are two ways of resolving this if you’re in my shoes - or 
in a role where the primary focus is not openstack developers or builders; the 
first is you coordinate work focusing on developer & end user experience 
upstream, working with openstack and the teams to come up with something that 
benefits everyone, or, you fork, and build the openstack equivalent to boto / 
awscli so you can provide a unified “one obvious way to consume openstack 
clouds”.

jesse

Regards,
Alexei




2014/1/21 Sean Dague mailto:s...@dague.net>>
On 01/21/2014 11:54 AM, Renat Akhmerov wrote:
>
> On 17 Jan 2014, at 22:00, Jamie Lennox 
> mailto:jamielen...@redhat.com>
> >> wrote:
>
>> (I don't buy the problem with large amounts of dependencies, if you
>> have a meta-package you just have one line in requirements and pip
>> will figure the rest out.)
>
> +1
>
> Renat Akhmerov
> @ Mirantis Inc.

Man, where were you then when we had to spend 3 weeks unwinding global
requirements in the gate because pip was figuring it out all kinds of
wrong, and we'd do things like uninstall and reinstall
python-keystoneclient 6 times during an install. Because after that
experience, I'm very anti "pip will figure the rest out".

Because it won't, not in python, where we're talking about libraries
that are in the global namespace, where python can only have 1 version
of a dependency installed.

If the the solution is every openstack project should install a venv for
all it's dependencies to get around this issue, then we're talking a
different problem (and a different architecture from what we've been
trying to do). But I find the idea of having 12 copies of
python-keystone client installed on my openstack environment to be
distasteful.

So come spend a month working on requirements updates in OpenStack
gate... and if you still believe "pip will figure it out", well you are
a braver man than I. :)

-Sean

--
Sean Dague
Samsung Research America
s...@dague.net / 
sean.da...@samsung.com
http://dague.net


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

I would like to propose to use this thread to gather and discuss software
requirements that our clients should meet.
Later we'll summarize all the requirements and use them during our work of
improving the clients.
By reaching listed requirements we'll be able to evaluate the success of
our refactoring.

To start this discussion:
* API versioning should be encapsulated inside of the client.

>From the end user perspective using the code should look something like:

import openstack.nova

nova = openstack.nova.create_client(...params TBD...)
nova.start_server(...)

Reasoning for this requirement:
If user would have to specify version explicitly his code will only work
with specific version of client/api or user would have to write several
code paths for different api versions.

Implementation:
We will have an interface (abstact class) that would contain all the
methods available in latest API.
Different versions of the API would implement this methods. In case method
cannot be implemented with existing API it should raise
"UnsupportedOperation" exception.

Glossary:
user - developer who uses the client/openstack API

Please share your thoughts about this requirement/other requirements.

Looking forward for your replies,
Alexei Kornienko
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

Hello,

I would like to end this requirements talk cause it doesn't make any sense
in term of python clients.
Initially the discussion was about "many clients projects with separate
requirements" VS "single client project with single requirements list".

At that moment we should have stop and actually open requirements lists for
python clients.
Basically all clients have the same requirement (cause they all do the same
stuff - sending HTTP requests K.O.)
There is absolutely no difference in the situation of many clients vs
single client.

Answering another question about "user only needs X (keystone) and we
install package with clients for all openstack services":
Size of keystone client (and any other client I suppose) is ~300Kb I don't
think that it's a big difference for the user to install package that is
~300Kb or ~10Mb (unless we are using openstack from Android).

>From the user perspective I think it's much easier to use client with
"everything included" rather than try to google for client package for some
rarely used service.

Regards,
Alexei




2014/1/21 Sean Dague 

> On 01/21/2014 11:54 AM, Renat Akhmerov wrote:
> >
> > On 17 Jan 2014, at 22:00, Jamie Lennox  > > wrote:
> >
> >> (I don't buy the problem with large amounts of dependencies, if you
> >> have a meta-package you just have one line in requirements and pip
> >> will figure the rest out.)
> >
> > +1
> >
> > Renat Akhmerov
> > @ Mirantis Inc.
>
> Man, where were you then when we had to spend 3 weeks unwinding global
> requirements in the gate because pip was figuring it out all kinds of
> wrong, and we'd do things like uninstall and reinstall
> python-keystoneclient 6 times during an install. Because after that
> experience, I'm very anti "pip will figure the rest out".
>
> Because it won't, not in python, where we're talking about libraries
> that are in the global namespace, where python can only have 1 version
> of a dependency installed.
>
> If the the solution is every openstack project should install a venv for
> all it's dependencies to get around this issue, then we're talking a
> different problem (and a different architecture from what we've been
> trying to do). But I find the idea of having 12 copies of
> python-keystone client installed on my openstack environment to be
> distasteful.
>
> So come spend a month working on requirements updates in OpenStack
> gate... and if you still believe "pip will figure it out", well you are
> a braver man than I. :)
>
> -Sean
>
> --
> Sean Dague
> Samsung Research America
> s...@dague.net / sean.da...@samsung.com
> http://dague.net
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Sean Dague]

On 01/21/2014 11:54 AM, Renat Akhmerov wrote:
> 
> On 17 Jan 2014, at 22:00, Jamie Lennox  > wrote:
> 
>> (I don't buy the problem with large amounts of dependencies, if you
>> have a meta-package you just have one line in requirements and pip
>> will figure the rest out.)
> 
> +1
> 
> Renat Akhmerov
> @ Mirantis Inc.

Man, where were you then when we had to spend 3 weeks unwinding global
requirements in the gate because pip was figuring it out all kinds of
wrong, and we'd do things like uninstall and reinstall
python-keystoneclient 6 times during an install. Because after that
experience, I'm very anti "pip will figure the rest out".

Because it won't, not in python, where we're talking about libraries
that are in the global namespace, where python can only have 1 version
of a dependency installed.

If the the solution is every openstack project should install a venv for
all it's dependencies to get around this issue, then we're talking a
different problem (and a different architecture from what we've been
trying to do). But I find the idea of having 12 copies of
python-keystone client installed on my openstack environment to be
distasteful.

So come spend a month working on requirements updates in OpenStack
gate... and if you still believe "pip will figure it out", well you are
a braver man than I. :)

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

On 18 Jan 2014, at 07:48, Sean Dague  wrote:

> I also think auto generated clients have a lot of challenges in the same
> way that full javascript pages in browsers have. If you screw up in a
> subtle way you can just completely disable your clients from connecting
> to your server entirely (or block them from using bits of the lesser
> used function path because a minor bit of schema fat fingered). So
> without a ton of additional verification on that path, I wouldn't want
> that anywhere near openstack. Especially with vendor extensions, which
> mean that enabling a network driver might break all your clients.

Yes, this makes tons of sense. The more I think about it the more I come to a 
conclusion that it will never work well enough (more organizationally though 
than technically). Additionally, there can be not 1 to 1 match between API 
methods and python code. For example, we’ve been considering some improvements 
in Mistral client library so that we can make in some cases a series of calls 
without actually hitting the server therefore implementing a masking behavior 
(in some scenarios it seems to make a lot of sense). Typically I’m talking 
about building a complex object from small pieces. We haven’t made a decision 
though. We’ll be discussing it with the community.

Renat Akhmerov
@ Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 21, 2014, at 10:54 AM, Renat Akhmerov 
mailto:rakhme...@mirantis.com>> wrote:


On 17 Jan 2014, at 22:00, Jamie Lennox 
mailto:jamielen...@redhat.com>> wrote:

(I don't buy the problem with large amounts of dependencies, if you have a 
meta-package you just have one line in requirements and pip will figure the 
rest out.)

+1

Renat Akhmerov
@ Mirantis Inc.

Do you use any other platform than Linux? Even donald - one of the python 
packaging leads and PyPI leads said this is a bad end-user experience for 
consumers of openstack clouds.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

On 17 Jan 2014, at 22:06, Robert Collins  wrote:

> On 17 January 2014 09:22, Renat Akhmerov  wrote:
>> Since it’s pretty easy to get lost among all the opinions I’d like to
>> clarify/ask a couple of things:
>> 
>> Keeping all the clients physically separate/combining them in to a single
>> library. Two things here:
>> 
>> In case of combining them, what exact project are we considering? If this
>> list is limited to core projects like nova and keystone what policy could we
>> have for other projects to join this list? (Incubation, graduation,
>> something else?)
>> In terms of granularity and easiness of development I’m for keeping them
>> separate but have them use the same boilerplate code, basically we need a
>> OpenStack Rest Client Framework which is flexible enough to address all the
>> needs in an abstract domain agnostic manner. I would assume that combining
>> them would be an additional organizational burden that every stakeholder
>> would have to deal with.
>> 
>> Has anyone ever considered an idea of generating a fully functional REST
>> client automatically based on an API specification (WADL could be used for
>> that)? Not sure how convenient it would be, it really depends on a
>> particular implementation, but as an idea it could be at least thought of.
>> Sounds a little bit crazy though, I recognize it :).
> 
> Launchpadlib which builds on wadllib did *exactly* that. It worked
> fairly well with the one caveat that it fell into the ORM trap - just
> in time lookups for everything with crippling roundtrips.
> 

Thanks, I’ll have a look at it.


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

On 17 Jan 2014, at 22:00, Jamie Lennox  wrote:

> (I don't buy the problem with large amounts of dependencies, if you have a 
> meta-package you just have one line in requirements and pip will figure the 
> rest out.)


+1

Renat Akhmerov
@ Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Sean Dague]

On 01/19/2014 11:50 PM, Jesse Noller wrote:
> 
> On Jan 19, 2014, at 5:37 PM, Jamie Lennox  > wrote:
> 
>> On Sat, 2014-01-18 at 09:13 -0500, Doug Hellmann wrote:
>>> I like the idea of a fresh start, but I don't think that's
>>> incompatible with the other work to clean up the existing clients.
>>> That cleanup work could help with creating the backwards compatibility
>>> layer, if a new library needs to include one, for example.
>>>
>>>
>>> As far as namespace packages and separate client libraries, I'm torn.
>>> It makes sense, and I originally assumed we would want to take that
>>> approach. The more I think about it, though, the more I like the
>>> approach Dean took with the CLI, creating a single repository with a
>>> team responsible for managing consistency in the UI.
>>>
>>>
>>> Doug
>>
>> This *is* the approach Dean took with the CLI. Have a package that
>> provides the CLI but then have the actual work handed off to the
>> individual clients (with quite a lot of glue).
> 
> And I think many of us are making the argument (or trying to) that the
> “a lot of glue” approach is wrong and unsustainable for both a unified
> CLI long term *and especially* for application developers.

100% agree. At some point take a look at the tempest rest client, and
you can see how entirely crazy different the APIs are between services -
https://github.com/openstack/tempest/blob/master/tempest/common/rest_client.py#L506

(The Tempest client is in no way a paragon of virtues, but by writing
our own client we've really discovered how lumpy this API is).

So I'm highly supportive of taking all the clients into a single
separate program which would produce the official python SDK, as well as
a unified CLI for OpenStack. The server programs should just be
producing server stacks, that end with the API. I think that would
empower a set of people that were most concerned with operator and
developer UX to be able to look at OpenStack as a whole.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 19, 2014, at 5:37 PM, Jamie Lennox 
mailto:jamielen...@redhat.com>> wrote:

On Sat, 2014-01-18 at 09:13 -0500, Doug Hellmann wrote:
I like the idea of a fresh start, but I don't think that's
incompatible with the other work to clean up the existing clients.
That cleanup work could help with creating the backwards compatibility
layer, if a new library needs to include one, for example.


As far as namespace packages and separate client libraries, I'm torn.
It makes sense, and I originally assumed we would want to take that
approach. The more I think about it, though, the more I like the
approach Dean took with the CLI, creating a single repository with a
team responsible for managing consistency in the UI.


Doug

This *is* the approach Dean took with the CLI. Have a package that
provides the CLI but then have the actual work handed off to the
individual clients (with quite a lot of glue).

And I think many of us are making the argument (or trying to) that the “a lot 
of glue” approach is wrong and unsustainable for both a unified CLI long term 
*and especially* for application developers.



On Sat, Jan 18, 2014 at 1:00 AM, Jamie Lennox 
mailto:jamielen...@redhat.com>>
wrote:
   I can't see any reason that all of these situations can't be
   met.

   We can finally take the openstack pypi namespace, move
   keystoneclient -> openstack.keystone and similar for the other
   projects. Have them all based upon openstack.base and probably
   an openstack.transport for transport.

   For the all-in-one users we can then just have
   openstack.client which depends on all of the openstack.x
   projects. This would satisfy the requirement of keeping
   projects seperate, but having the one entry point for newer
   users. Similar to the OSC project (which could acutally rely
   on the new all-in-one).

   This would also satisfy a lot of the clients who have i know
   are looking to move to a version 2 and break compatability
   with some of the crap from the early days.

   I think what is most important here is deciding what we want
   from our clients and discussing a common base that we are
   happy to support - not just renaming the existing ones.

   (I don't buy the problem with large amounts of dependencies,
   if you have a meta-package you just have one line in
   requirements and pip will figure the rest out.)

   Jamie

   - Original Message -
From: "Jonathan LaCour" 
mailto:jonathan-li...@cleverdevil.org>>
To: "OpenStack Development Mailing List (not for usage
   questions)" 
mailto:openstack-dev@lists.openstack.org>>
Sent: Saturday, 18 January, 2014 4:00:58 AM
Subject: Re: [openstack-dev] a "common" client library


On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft <
   don...@stufft.io<mailto:don...@stufft.io> > wrote:




On Jan 16, 2014, at 4:06 PM, Jesse Noller <
   jesse.nol...@rackspace.com<mailto:jesse.nol...@rackspace.com> >
wrote:





On Jan 16, 2014, at 2:22 PM, Renat Akhmerov <
   rakhme...@mirantis.com<mailto:rakhme...@mirantis.com> > wrote:




Since it’s pretty easy to get lost among all the opinions
   I’d like to
clarify/ask a couple of things:




   * Keeping all the clients physically separate/combining
   them in to a
   single library. Two things here:

   * In case of combining them, what exact project are
   we considering?
   If this list is limited to core projects like nova
   and keystone what
   policy could we have for other projects to join this
   list?
   (Incubation, graduation, something else?)

   * In terms of granularity and easiness of
   development I’m for keeping
   them separate but have them use the same boilerplate
   code, basically
   we need a OpenStack Rest Client Framework which is
   flexible enough
   to address all the needs in an abstract domain
   agnostic manner. I
   would assume that combining them would be an
   additional
   organizational burden that every stakeholder would
   have to deal
   with.

Keeping them separate is awesome for *us* but really,
   really, really sucks
for users trying to use the system.

I agree. Keeping them separate trades user usability for
   developer usability,
I think user usability is a better thing to strive for.
100% agree with this. In order for OpenStack to be its most
   successful, I
believe firmly that a focus on end-users and deployers needs
   to take the
forefront. That means making OpenStack clouds as easy to
   consume/leverage as
possible for users and tool builders, and
   simplifying/streamlining as much
as possible.

I think that a single, common client project, based upon
   package namespaces,
with a unified, cohesive feel is a big step 

Re: [openstack-dev] a "common" client library

[Jamie Lennox]

On Sat, 2014-01-18 at 09:13 -0500, Doug Hellmann wrote:
> I like the idea of a fresh start, but I don't think that's
> incompatible with the other work to clean up the existing clients.
> That cleanup work could help with creating the backwards compatibility
> layer, if a new library needs to include one, for example.
> 
> 
> As far as namespace packages and separate client libraries, I'm torn.
> It makes sense, and I originally assumed we would want to take that
> approach. The more I think about it, though, the more I like the
> approach Dean took with the CLI, creating a single repository with a
> team responsible for managing consistency in the UI.
> 
> 
> Doug

This *is* the approach Dean took with the CLI. Have a package that
provides the CLI but then have the actual work handed off to the
individual clients (with quite a lot of glue).

> 
> On Sat, Jan 18, 2014 at 1:00 AM, Jamie Lennox 
> wrote:
> I can't see any reason that all of these situations can't be
> met.
> 
> We can finally take the openstack pypi namespace, move
> keystoneclient -> openstack.keystone and similar for the other
> projects. Have them all based upon openstack.base and probably
> an openstack.transport for transport.
> 
> For the all-in-one users we can then just have
> openstack.client which depends on all of the openstack.x
> projects. This would satisfy the requirement of keeping
> projects seperate, but having the one entry point for newer
> users. Similar to the OSC project (which could acutally rely
> on the new all-in-one).
> 
> This would also satisfy a lot of the clients who have i know
> are looking to move to a version 2 and break compatability
> with some of the crap from the early days.
> 
> I think what is most important here is deciding what we want
> from our clients and discussing a common base that we are
> happy to support - not just renaming the existing ones.
> 
> (I don't buy the problem with large amounts of dependencies,
> if you have a meta-package you just have one line in
> requirements and pip will figure the rest out.)
> 
> Jamie
> 
> - Original Message -
> > From: "Jonathan LaCour" 
>     > To: "OpenStack Development Mailing List (not for usage
> questions)" 
> > Sent: Saturday, 18 January, 2014 4:00:58 AM
> > Subject: Re: [openstack-dev] a "common" client library
> >
> 
> > On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft <
> don...@stufft.io > wrote:
> >
> >
> >
> >
> > On Jan 16, 2014, at 4:06 PM, Jesse Noller <
> jesse.nol...@rackspace.com >
> > wrote:
> >
> >
> >
> >
> >
> > On Jan 16, 2014, at 2:22 PM, Renat Akhmerov <
> rakhme...@mirantis.com > wrote:
> >
> >
> >
> >
> > Since it’s pretty easy to get lost among all the opinions
> I’d like to
> > clarify/ask a couple of things:
> >
> >
> >
> 
> > * Keeping all the clients physically separate/combining
> them in to a
> > single library. Two things here:
> 
> > * In case of combining them, what exact project are
> we considering?
> > If this list is limited to core projects like nova
> and keystone what
> > policy could we have for other projects to join this
> list?
> > (Incubation, graduation, something else?)
> 
> > * In terms of granularity and easiness of
> development I’m for keeping
> > them separate but have them use the same boilerplate
> code, basically
> > we need a OpenStack Rest Client Framework which is
> flexible enough
> > to address all the needs in an abstract domain
> agnostic manner. I
> > would assume that combining them would be an
> additional
> > organizational burden that every stakeholder would
> have to deal
> > with.
> >
> > Keeping them separate is awesome for *us* but really,
> 

Re: [openstack-dev] a "common" client library

[Robert Collins]

On 19 January 2014 04:48, Sean Dague  wrote:
> On 01/18/2014 01:06 AM, Robert Collins wrote:

>> Launchpadlib which builds on wadllib did *exactly* that. It worked
>> fairly well with the one caveat that it fell into the ORM trap - just
>> in time lookups for everything with crippling roundtrips.
>
> -1 if the answer to anything is WADL. It's terrible, and an abandoned
> sun rfc at this point. We've got some real progress in getting JSON
> schema in place in Nova (for validation, but it's incremental steps, and
> good ones), which I think is much more fruitful.
>
> I also think auto generated clients have a lot of challenges in the same
> way that full javascript pages in browsers have. If you screw up in a
> subtle way you can just completely disable your clients from connecting
> to your server entirely (or block them from using bits of the lesser
> used function path because a minor bit of schema fat fingered). So
> without a ton of additional verification on that path, I wouldn't want
> that anywhere near openstack. Especially with vendor extensions, which
> mean that enabling a network driver might break all your clients.

To be clear: I'm not advocating this approach. Just answering the
question 'has it been tried', with 'yes, and this is what was wrong'.

The actual code generation and execution worked very well, and most
features added to the server side APIs were available immediately in
the client with no upgrades. However any new /type/ of feature
required changes to the compiler.

Personally, I think hand crafted clients have a much better feel to
them - idiomatic to the language they are in, easier to predict, and
much easier to understand.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Donald Stufft]

On Jan 18, 2014, at 9:58 AM, Jesse Noller  wrote:

> 
> On Jan 18, 2014, at 12:00 AM, Jamie Lennox  wrote:
> 
>> I can't see any reason that all of these situations can't be met. 
>> 
>> We can finally take the openstack pypi namespace, move keystoneclient -> 
>> openstack.keystone and similar for the other projects. Have them all based 
>> upon openstack.base and probably an openstack.transport for transport.
>> 
>> For the all-in-one users we can then just have openstack.client which 
>> depends on all of the openstack.x projects. This would satisfy the 
>> requirement of keeping projects seperate, but having the one entry point for 
>> newer users. Similar to the OSC project (which could acutally rely on the 
>> new all-in-one).
>> 
>> This would also satisfy a lot of the clients who have i know are looking to 
>> move to a version 2 and break compatability with some of the crap from the 
>> early days.
>> 
>> I think what is most important here is deciding what we want from our 
>> clients and discussing a common base that we are happy to support - not just 
>> renaming the existing ones.
>> 
>> (I don't buy the problem with large amounts of dependencies, if you have a 
>> meta-package you just have one line in requirements and pip will figure the 
>> rest out.)
> 
> You’re assuming:
> 
> 1: Pip works when installing the entire dependency graph (it often doesn’t)
> 2: For some of these requirements, the user has a compiler installed (they 
> don’t)
> 3: Installing 1 “meta package” that install N+K dependencies makes end user 
> consumers happy (it doesn’t)
> 4: All of these dependencies make shipping a single binary deployment easier 
> (it doesn’t)
> 5: Installing and using all of these things makes using openstack within my 
> code conceptually simpler (it doesn’t)
> 
> We can start with *not* renaming the sub clients (meaning) collapsing them 
> into the singular namespace; but the problem is that every one of those sub 
> dependencies is potential liability to someone using this single client. 
> 
> If yes, we could only target fedora, and rely on yum & rpm, I’d agree with 
> you - but for python application dependencies across multiple OSes and 
> developers doing ci/cd using these systems I can’t. I also don’t want user to 
> stumble into the nuanced vagaries of the sub-clients when writing application 
> code; writing glue code to bind them all together does work very well (we 
> know this from experience).
> 

As much as I would like to say (with my pip developer and PyPI Admin hat on) 
that depending on 22+ libraries in a single client will be a seamless 
experience for end users I can’t in good faith say that it would be yet. We’re 
working on trying to make that true but honestly each dependency in a graph 
does introduce risk.

As of right now there is no real dependency solver in pip, so if someone 
depends on the openstack client themselves, and then depends on something else 
that depends on one of the sub clients as well if those version specs don’t 
match up there is a very good chance that the end user will run into a very 
confusing message at runtime. Openstack itself has run into this problem and it 
was a big motivator for the global requirements project.

Additionally it’s not uncommon for users to have policy driven requirements 
that require them to get every dependency they pull in checked for compliance 
(License, security etc). Having a 22+ node dependency graph makes this issue 
much harder in general.

I also believe in general it’s asking for user confusion. It’s much simpler to 
document a single way of doing it, however splitting the clients up and then 
wrapping them with a single “openstack” client means that you have at least two 
ways of doing something; The direct “use just a single library” approach and 
the “use the openstack wrapper” approach. Don’t underestimate the confusion 
this will cause end users.

Keeping them all under one project will make it far easier to have a cohesive 
API amongst all the various services, it will reduce duplication of efforts, as 
well as make it easier to track security updates and I believe a wholly 
superior end user experience.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Donald Stufft]

On Jan 18, 2014, at 12:58 AM, Robert Collins  wrote:

> Out of interest - whats the overhead of running tls compression
> against compressed data? Is it really noticable?

The overhead doesn’t really matter much as you want TLS
Compression disabled because of CRIME anyways. Most Linux
distros and such ship with it disabled by default now IIRC.

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Sean Dague]

On 01/18/2014 01:06 AM, Robert Collins wrote:
> On 17 January 2014 09:22, Renat Akhmerov  wrote:
>> Since it’s pretty easy to get lost among all the opinions I’d like to
>> clarify/ask a couple of things:
>>
>> Keeping all the clients physically separate/combining them in to a single
>> library. Two things here:
>>
>> In case of combining them, what exact project are we considering? If this
>> list is limited to core projects like nova and keystone what policy could we
>> have for other projects to join this list? (Incubation, graduation,
>> something else?)
>> In terms of granularity and easiness of development I’m for keeping them
>> separate but have them use the same boilerplate code, basically we need a
>> OpenStack Rest Client Framework which is flexible enough to address all the
>> needs in an abstract domain agnostic manner. I would assume that combining
>> them would be an additional organizational burden that every stakeholder
>> would have to deal with.
>>
>> Has anyone ever considered an idea of generating a fully functional REST
>> client automatically based on an API specification (WADL could be used for
>> that)? Not sure how convenient it would be, it really depends on a
>> particular implementation, but as an idea it could be at least thought of.
>> Sounds a little bit crazy though, I recognize it :).
> 
> Launchpadlib which builds on wadllib did *exactly* that. It worked
> fairly well with the one caveat that it fell into the ORM trap - just
> in time lookups for everything with crippling roundtrips.

-1 if the answer to anything is WADL. It's terrible, and an abandoned
sun rfc at this point. We've got some real progress in getting JSON
schema in place in Nova (for validation, but it's incremental steps, and
good ones), which I think is much more fruitful.

I also think auto generated clients have a lot of challenges in the same
way that full javascript pages in browsers have. If you screw up in a
subtle way you can just completely disable your clients from connecting
to your server entirely (or block them from using bits of the lesser
used function path because a minor bit of schema fat fingered). So
without a ton of additional verification on that path, I wouldn't want
that anywhere near openstack. Especially with vendor extensions, which
mean that enabling a network driver might break all your clients.

-Sean

-- 
Sean Dague
Samsung Research America
s...@dague.net / sean.da...@samsung.com
http://dague.net



signature.asc
Description: OpenPGP digital signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 18, 2014, at 12:00 AM, Jamie Lennox  wrote:

> I can't see any reason that all of these situations can't be met. 
> 
> We can finally take the openstack pypi namespace, move keystoneclient -> 
> openstack.keystone and similar for the other projects. Have them all based 
> upon openstack.base and probably an openstack.transport for transport.
> 
> For the all-in-one users we can then just have openstack.client which depends 
> on all of the openstack.x projects. This would satisfy the requirement of 
> keeping projects seperate, but having the one entry point for newer users. 
> Similar to the OSC project (which could acutally rely on the new all-in-one).
> 
> This would also satisfy a lot of the clients who have i know are looking to 
> move to a version 2 and break compatability with some of the crap from the 
> early days.
> 
> I think what is most important here is deciding what we want from our clients 
> and discussing a common base that we are happy to support - not just renaming 
> the existing ones.
> 
> (I don't buy the problem with large amounts of dependencies, if you have a 
> meta-package you just have one line in requirements and pip will figure the 
> rest out.)

You’re assuming:

1: Pip works when installing the entire dependency graph (it often doesn’t)
2: For some of these requirements, the user has a compiler installed (they 
don’t)
3: Installing 1 “meta package” that install N+K dependencies makes end user 
consumers happy (it doesn’t)
4: All of these dependencies make shipping a single binary deployment easier 
(it doesn’t)
5: Installing and using all of these things makes using openstack within my 
code conceptually simpler (it doesn’t)

We can start with *not* renaming the sub clients (meaning) collapsing them into 
the singular namespace; but the problem is that every one of those sub 
dependencies is potential liability to someone using this single client. 

If yes, we could only target fedora, and rely on yum & rpm, I’d agree with you 
- but for python application dependencies across multiple OSes and developers 
doing ci/cd using these systems I can’t. I also don’t want user to stumble into 
the nuanced vagaries of the sub-clients when writing application code; writing 
glue code to bind them all together does work very well (we know this from 
experience).


> 
> Jamie
> 
> - Original Message -
>> From: "Jonathan LaCour" 
>> To: "OpenStack Development Mailing List (not for usage questions)" 
>> 
>> Sent: Saturday, 18 January, 2014 4:00:58 AM
>> Subject: Re: [openstack-dev] a "common" client library
>> 
>> On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft < don...@stufft.io > wrote:
>> 
>> 
>> 
>> 
>> On Jan 16, 2014, at 4:06 PM, Jesse Noller < jesse.nol...@rackspace.com >
>> wrote:
>> 
>> 
>> 
>> 
>> 
>> On Jan 16, 2014, at 2:22 PM, Renat Akhmerov < rakhme...@mirantis.com > wrote:
>> 
>> 
>> 
>> 
>> Since it’s pretty easy to get lost among all the opinions I’d like to
>> clarify/ask a couple of things:
>> 
>> 
>> 
>>* Keeping all the clients physically separate/combining them in to a
>>single library. Two things here:
>>* In case of combining them, what exact project are we considering?
>>If this list is limited to core projects like nova and keystone what
>>policy could we have for other projects to join this list?
>>(Incubation, graduation, something else?)
>>* In terms of granularity and easiness of development I’m for keeping
>>them separate but have them use the same boilerplate code, basically
>>we need a OpenStack Rest Client Framework which is flexible enough
>>to address all the needs in an abstract domain agnostic manner. I
>>would assume that combining them would be an additional
>>organizational burden that every stakeholder would have to deal
>>with.
>> 
>> Keeping them separate is awesome for *us* but really, really, really sucks
>> for users trying to use the system.
>> 
>> I agree. Keeping them separate trades user usability for developer usability,
>> I think user usability is a better thing to strive for.
>> 100% agree with this. In order for OpenStack to be its most successful, I
>> believe firmly that a focus on end-users and deployers needs to take the
>> forefront. That means making OpenStack clouds as easy to consume/leverage as
>> possible for users and tool builders, and simplifying/streamlining as much
>> as possible.
>> 
>> I think that a single, common client project, based u

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 18, 2014, at 8:13 AM, Doug Hellmann 
mailto:doug.hellm...@dreamhost.com>> wrote:

I like the idea of a fresh start, but I don't think that's incompatible with 
the other work to clean up the existing clients. That cleanup work could help 
with creating the backwards compatibility layer, if a new library needs to 
include one, for example.

As far as namespace packages and separate client libraries, I'm torn. It makes 
sense, and I originally assumed we would want to take that approach. The more I 
think about it, though, the more I like the approach Dean took with the CLI, 
creating a single repository with a team responsible for managing consistency 
in the UI.

Doug



I’m going to pursue the latter - but at the same time the current effort to 
clean things up seems to be running afoul of the keystone client changes in 
flight?


On Sat, Jan 18, 2014 at 1:00 AM, Jamie Lennox 
mailto:jamielen...@redhat.com>> wrote:
I can't see any reason that all of these situations can't be met.

We can finally take the openstack pypi namespace, move keystoneclient -> 
openstack.keystone and similar for the other projects. Have them all based upon 
openstack.base and probably an openstack.transport for transport.

For the all-in-one users we can then just have openstack.client which depends 
on all of the openstack.x projects. This would satisfy the requirement of 
keeping projects seperate, but having the one entry point for newer users. 
Similar to the OSC project (which could acutally rely on the new all-in-one).

This would also satisfy a lot of the clients who have i know are looking to 
move to a version 2 and break compatability with some of the crap from the 
early days.

I think what is most important here is deciding what we want from our clients 
and discussing a common base that we are happy to support - not just renaming 
the existing ones.

(I don't buy the problem with large amounts of dependencies, if you have a 
meta-package you just have one line in requirements and pip will figure the 
rest out.)

Jamie

- Original Message -
> From: "Jonathan LaCour" 
> mailto:jonathan-li...@cleverdevil.org>>
> To: "OpenStack Development Mailing List (not for usage questions)" 
> mailto:openstack-dev@lists.openstack.org>>
> Sent: Saturday, 18 January, 2014 4:00:58 AM
> Subject: Re: [openstack-dev] a "common" client library
>
> On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft < 
> don...@stufft.io<mailto:don...@stufft.io> > wrote:
>
>
>
>
> On Jan 16, 2014, at 4:06 PM, Jesse Noller < 
> jesse.nol...@rackspace.com<mailto:jesse.nol...@rackspace.com> >
> wrote:
>
>
>
>
>
> On Jan 16, 2014, at 2:22 PM, Renat Akhmerov < 
> rakhme...@mirantis.com<mailto:rakhme...@mirantis.com> > wrote:
>
>
>
>
> Since it’s pretty easy to get lost among all the opinions I’d like to
> clarify/ask a couple of things:
>
>
>
> * Keeping all the clients physically separate/combining them in to a
> single library. Two things here:
> * In case of combining them, what exact project are we considering?
> If this list is limited to core projects like nova and keystone what
> policy could we have for other projects to join this list?
> (Incubation, graduation, something else?)
> * In terms of granularity and easiness of development I’m for keeping
> them separate but have them use the same boilerplate code, basically
> we need a OpenStack Rest Client Framework which is flexible enough
> to address all the needs in an abstract domain agnostic manner. I
> would assume that combining them would be an additional
> organizational burden that every stakeholder would have to deal
> with.
>
> Keeping them separate is awesome for *us* but really, really, really sucks
> for users trying to use the system.
>
> I agree. Keeping them separate trades user usability for developer usability,
> I think user usability is a better thing to strive for.
> 100% agree with this. In order for OpenStack to be its most successful, I
> believe firmly that a focus on end-users and deployers needs to take the
> forefront. That means making OpenStack clouds as easy to consume/leverage as
> possible for users and tool builders, and simplifying/streamlining as much
> as possible.
>
> I think that a single, common client project, based upon package namespaces,
> with a unified, cohesive feel is a big step in this direction.
>
> --
> Jonathan LaCour
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 17, 2014, at 10:03 PM, John Utz  wrote:

> Outlook Web MUA, forgive the top post. :-(
> 
> While a single import line that brings all the good stuff in at one shot is 
> very convenient for the creation of an application, it would muddy the 
> security picture *substantially* for the exact type of developer\customer 
> that you would be targeting with this sort of syntactic sugar.
> 
> As Jesse alludes to below, the expanding tree of dependencies would be masked 
> by the aggregation.
> 
> So, most likely, they would be pulling in vast numbers of things that they 
> don't require to get their simple app done (there's an idea! an eclipse 
> plugin that helpfully points out all the things that you are *not* using and 
> offers to redo your imports for you :-) ).
> 
> As a result, when a security defect is published concerning one of those 
> hidden dependencies, they will not have any reason to think that it effects 
> them.
> 
> just my us$0.02;
> 


Thats why when a security defect in the *client side* tools is announced it 
affects the entire client library which is then versioned. Not just a sub part: 
you re-ship the entire thing. 

This is how every other SDK *other* than openstack’s cli tools handles this.



> johnu 
> 
> From: Jesse Noller [jesse.nol...@rackspace.com]
> Sent: Thursday, January 16, 2014 5:42 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Cc: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] a "common" client library
> 
> On Jan 16, 2014, at 4:59 PM, "Renat Akhmerov" 
> mailto:rakhme...@mirantis.com>> wrote:
> 
> On 16 Jan 2014, at 13:06, Jesse Noller 
> mailto:jesse.nol...@rackspace.com>> wrote:
> 
> Since it’s pretty easy to get lost among all the opinions I’d like to 
> clarify/ask a couple of things:
> 
> 
>  *   Keeping all the clients physically separate/combining them in to a 
> single library. Two things here:
> *   In case of combining them, what exact project are we considering? If 
> this list is limited to core projects like nova and keystone what policy 
> could we have for other projects to join this list? (Incubation, graduation, 
> something else?)
> *   In terms of granularity and easiness of development I’m for keeping 
> them separate but have them use the same boilerplate code, basically we need 
> a OpenStack Rest Client Framework which is flexible enough to address all the 
> needs in an abstract domain agnostic manner. I would assume that combining 
> them would be an additional organizational burden that every stakeholder 
> would have to deal with.
> 
> Keeping them separate is awesome for *us* but really, really, really sucks 
> for users trying to use the system.
> 
> You may be right but not sure that adding another line into requirements.txt 
> is a huge loss of usability.
> 
> 
> It is when that 1 dependency pulls in 6 others that pull in 10 more - every 
> little barrier or potential failure from the inability to make a static 
> binary to how each tool acts different is a paper cut of frustration to an 
> end user.
> 
> Most of the time the clients don't even properly install because of 
> dependencies on setuptools plugins and other things. For developers (as I've 
> said) the story is worse: you have potentially 22+ individual packages and 
> their dependencies to deal with if they want to use a complete openstack 
> install from their code.
> 
> So it doesn't boil down to just 1 dependency: it's a long laundry list of 
> things that make consumers' lives more difficult and painful.
> 
> This doesn't even touch on the fact there aren't blessed SDKs or tools 
> pointing users to consume openstack in their preferred programming language.
> 
> Shipping an API isn't enough - but it can be fixed easily enough.
> 
> Renat Akhmerov
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Doug Hellmann]

I like the idea of a fresh start, but I don't think that's incompatible
with the other work to clean up the existing clients. That cleanup work
could help with creating the backwards compatibility layer, if a new
library needs to include one, for example.

As far as namespace packages and separate client libraries, I'm torn. It
makes sense, and I originally assumed we would want to take that approach.
The more I think about it, though, the more I like the approach Dean took
with the CLI, creating a single repository with a team responsible for
managing consistency in the UI.

Doug


On Sat, Jan 18, 2014 at 1:00 AM, Jamie Lennox wrote:

> I can't see any reason that all of these situations can't be met.
>
> We can finally take the openstack pypi namespace, move keystoneclient ->
> openstack.keystone and similar for the other projects. Have them all based
> upon openstack.base and probably an openstack.transport for transport.
>
> For the all-in-one users we can then just have openstack.client which
> depends on all of the openstack.x projects. This would satisfy the
> requirement of keeping projects seperate, but having the one entry point
> for newer users. Similar to the OSC project (which could acutally rely on
> the new all-in-one).
>
> This would also satisfy a lot of the clients who have i know are looking
> to move to a version 2 and break compatability with some of the crap from
> the early days.
>
> I think what is most important here is deciding what we want from our
> clients and discussing a common base that we are happy to support - not
> just renaming the existing ones.
>
> (I don't buy the problem with large amounts of dependencies, if you have a
> meta-package you just have one line in requirements and pip will figure the
> rest out.)
>
> Jamie
>
> - Original Message -
> > From: "Jonathan LaCour" 
> > To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev@lists.openstack.org>
> > Sent: Saturday, 18 January, 2014 4:00:58 AM
> > Subject: Re: [openstack-dev] a "common" client library
> >
> > On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft < don...@stufft.io >
> wrote:
> >
> >
> >
> >
> > On Jan 16, 2014, at 4:06 PM, Jesse Noller < jesse.nol...@rackspace.com >
> > wrote:
> >
> >
> >
> >
> >
> > On Jan 16, 2014, at 2:22 PM, Renat Akhmerov < rakhme...@mirantis.com >
> wrote:
> >
> >
> >
> >
> > Since it’s pretty easy to get lost among all the opinions I’d like to
> > clarify/ask a couple of things:
> >
> >
> >
> > * Keeping all the clients physically separate/combining them in to a
> > single library. Two things here:
> > * In case of combining them, what exact project are we
> considering?
> > If this list is limited to core projects like nova and keystone
> what
> > policy could we have for other projects to join this list?
> > (Incubation, graduation, something else?)
> > * In terms of granularity and easiness of development I’m for
> keeping
> > them separate but have them use the same boilerplate code,
> basically
> > we need a OpenStack Rest Client Framework which is flexible
> enough
> > to address all the needs in an abstract domain agnostic manner. I
> > would assume that combining them would be an additional
> > organizational burden that every stakeholder would have to deal
> > with.
> >
> > Keeping them separate is awesome for *us* but really, really, really
> sucks
> > for users trying to use the system.
> >
> > I agree. Keeping them separate trades user usability for developer
> usability,
> > I think user usability is a better thing to strive for.
> > 100% agree with this. In order for OpenStack to be its most successful, I
> > believe firmly that a focus on end-users and deployers needs to take the
> > forefront. That means making OpenStack clouds as easy to
> consume/leverage as
> > possible for users and tool builders, and simplifying/streamlining as
> much
> > as possible.
> >
> > I think that a single, common client project, based upon package
> namespaces,
> > with a unified, cohesive feel is a big step in this direction.
> >
> > --
> > Jonathan LaCour
> >
> >
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Doug Hellmann]

On Fri, Jan 17, 2014 at 11:03 PM, John Utz  wrote:

> Outlook Web MUA, forgive the top post. :-(
>
> While a single import line that brings all the good stuff in at one shot
> is very convenient for the creation of an application, it would muddy the
> security picture *substantially* for the exact type of developer\customer
> that you would be targeting with this sort of syntactic sugar.
>
> As Jesse alludes to below, the expanding tree of dependencies would be
> masked by the aggregation.
>
> So, most likely, they would be pulling in vast numbers of things that they
> don't require to get their simple app done (there's an idea! an eclipse
> plugin that helpfully points out all the things that you are *not* using
> and offers to redo your imports for you :-) ).
>

I'm not sure what "vast numbers" of things you mean. The point is to make
one thing to talk to an OpenStack cloud consistently, instead of a separate
library for every facet of the cloud. Surely building on a common code
base will have the opposite effect you suggest -- it should reduce the
number of dependencies, and make it easier to track security updates in
those dependencies.

Doug



>
> As a result, when a security defect is published concerning one of those
> hidden dependencies, they will not have any reason to think that it effects
> them.
>
> just my us$0.02;
>
> johnu
> 
> From: Jesse Noller [jesse.nol...@rackspace.com]
> Sent: Thursday, January 16, 2014 5:42 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Cc: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] a "common" client library
>
> On Jan 16, 2014, at 4:59 PM, "Renat Akhmerov"  <mailto:rakhme...@mirantis.com>> wrote:
>
> On 16 Jan 2014, at 13:06, Jesse Noller  jesse.nol...@rackspace.com>> wrote:
>
> Since it’s pretty easy to get lost among all the opinions I’d like to
> clarify/ask a couple of things:
>
>
>   *   Keeping all the clients physically separate/combining them in to a
> single library. Two things here:
>  *   In case of combining them, what exact project are we considering?
> If this list is limited to core projects like nova and keystone what policy
> could we have for other projects to join this list? (Incubation,
> graduation, something else?)
>  *   In terms of granularity and easiness of development I’m for
> keeping them separate but have them use the same boilerplate code,
> basically we need a OpenStack Rest Client Framework which is flexible
> enough to address all the needs in an abstract domain agnostic manner. I
> would assume that combining them would be an additional organizational
> burden that every stakeholder would have to deal with.
>
> Keeping them separate is awesome for *us* but really, really, really sucks
> for users trying to use the system.
>
> You may be right but not sure that adding another line into
> requirements.txt is a huge loss of usability.
>
>
> It is when that 1 dependency pulls in 6 others that pull in 10 more -
> every little barrier or potential failure from the inability to make a
> static binary to how each tool acts different is a paper cut of frustration
> to an end user.
>
> Most of the time the clients don't even properly install because of
> dependencies on setuptools plugins and other things. For developers (as
> I've said) the story is worse: you have potentially 22+ individual packages
> and their dependencies to deal with if they want to use a complete
> openstack install from their code.
>
> So it doesn't boil down to just 1 dependency: it's a long laundry list of
> things that make consumers' lives more difficult and painful.
>
> This doesn't even touch on the fact there aren't blessed SDKs or tools
> pointing users to consume openstack in their preferred programming language.
>
> Shipping an API isn't enough - but it can be fixed easily enough.
>
> Renat Akhmerov
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org
> >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Robert Collins]

On 17 January 2014 09:22, Renat Akhmerov  wrote:
> Since it’s pretty easy to get lost among all the opinions I’d like to
> clarify/ask a couple of things:
>
> Keeping all the clients physically separate/combining them in to a single
> library. Two things here:
>
> In case of combining them, what exact project are we considering? If this
> list is limited to core projects like nova and keystone what policy could we
> have for other projects to join this list? (Incubation, graduation,
> something else?)
> In terms of granularity and easiness of development I’m for keeping them
> separate but have them use the same boilerplate code, basically we need a
> OpenStack Rest Client Framework which is flexible enough to address all the
> needs in an abstract domain agnostic manner. I would assume that combining
> them would be an additional organizational burden that every stakeholder
> would have to deal with.
>
> Has anyone ever considered an idea of generating a fully functional REST
> client automatically based on an API specification (WADL could be used for
> that)? Not sure how convenient it would be, it really depends on a
> particular implementation, but as an idea it could be at least thought of.
> Sounds a little bit crazy though, I recognize it :).

Launchpadlib which builds on wadllib did *exactly* that. It worked
fairly well with the one caveat that it fell into the ORM trap - just
in time lookups for everything with crippling roundtrips.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Robert Collins]

On 17 January 2014 08:03, Alexei Kornienko  wrote:
> Hello Joe,

> 2)Another option would be to follow waterfall process and create a solid
> library interface before including it to all client projects. However such
> approach this period can take unknown amount of time and can be easily
> failed during integration stage cause requirements change or some other
> reason.
>
> Please let me know what you think.

Given how fast OpenStack moves, I think waterfall is a non-starter.
I'd do a library from the start, and just do backwards compat - it's
really not that hard  :)

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Robert Collins]

On 17 January 2014 06:57, Mark Washenberger
 wrote:

> Just throwing this out there because it seems relevant to client design.
>
> As we've been looking at porting clients to using v2 of the Images API, its
> seems more and more to me that including the *server* version in the main
> import path is a real obstacle.
>
> IMO any future client libs should write library interfaces based on the
> peculiarities of user needs, not based on the vagaries of the server
> version. So as a user of this library I would do something like:
>
>   1 from openstack.api import images
>   2 client = images.make_me_a_client(auth_url, etcetera) # all version
> negotiation is happening here
>   3 client.list_images()  # works more or less same no matter who I'm
> talking to
>
> Now, there would still likely be hidden implementation code that is
> different per server version and which is instantiated in line 2 above, and
> maybe that's the library path stuff you are talking about.

That design seems guaranteed to behave somewhat poorly (e.g. fail to
upgrade) when servers are upgraded - for short lived processes like
'nova boot' that doesn't matter, but for software running in a daemon
- e.g. in nova-api talking to neutron - that seems much more likely to
be a problem.

I think the pseudo code is fine, but client shouldn't be a concrete
version locked client, rather a proxy object that can revalidate the
version every {sensible time period} and/or observe HTTP headers to
detect when upgrades are possible (or downgrades are required).

-Rob



-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jamie Lennox]

I can't see any reason that all of these situations can't be met. 

We can finally take the openstack pypi namespace, move keystoneclient -> 
openstack.keystone and similar for the other projects. Have them all based upon 
openstack.base and probably an openstack.transport for transport.

For the all-in-one users we can then just have openstack.client which depends 
on all of the openstack.x projects. This would satisfy the requirement of 
keeping projects seperate, but having the one entry point for newer users. 
Similar to the OSC project (which could acutally rely on the new all-in-one).

This would also satisfy a lot of the clients who have i know are looking to 
move to a version 2 and break compatability with some of the crap from the 
early days.

I think what is most important here is deciding what we want from our clients 
and discussing a common base that we are happy to support - not just renaming 
the existing ones.

(I don't buy the problem with large amounts of dependencies, if you have a 
meta-package you just have one line in requirements and pip will figure the 
rest out.)

Jamie

- Original Message -
> From: "Jonathan LaCour" 
> To: "OpenStack Development Mailing List (not for usage questions)" 
> 
> Sent: Saturday, 18 January, 2014 4:00:58 AM
> Subject: Re: [openstack-dev] a "common" client library
> 
> On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft < don...@stufft.io > wrote:
> 
> 
> 
> 
> On Jan 16, 2014, at 4:06 PM, Jesse Noller < jesse.nol...@rackspace.com >
> wrote:
> 
> 
> 
> 
> 
> On Jan 16, 2014, at 2:22 PM, Renat Akhmerov < rakhme...@mirantis.com > wrote:
> 
> 
> 
> 
> Since it’s pretty easy to get lost among all the opinions I’d like to
> clarify/ask a couple of things:
> 
> 
> 
> * Keeping all the clients physically separate/combining them in to a
> single library. Two things here:
> * In case of combining them, what exact project are we considering?
> If this list is limited to core projects like nova and keystone what
> policy could we have for other projects to join this list?
> (Incubation, graduation, something else?)
> * In terms of granularity and easiness of development I’m for keeping
> them separate but have them use the same boilerplate code, basically
> we need a OpenStack Rest Client Framework which is flexible enough
> to address all the needs in an abstract domain agnostic manner. I
> would assume that combining them would be an additional
> organizational burden that every stakeholder would have to deal
> with.
> 
> Keeping them separate is awesome for *us* but really, really, really sucks
> for users trying to use the system.
> 
> I agree. Keeping them separate trades user usability for developer usability,
> I think user usability is a better thing to strive for.
> 100% agree with this. In order for OpenStack to be its most successful, I
> believe firmly that a focus on end-users and deployers needs to take the
> forefront. That means making OpenStack clouds as easy to consume/leverage as
> possible for users and tool builders, and simplifying/streamlining as much
> as possible.
> 
> I think that a single, common client project, based upon package namespaces,
> with a unified, cohesive feel is a big step in this direction.
> 
> --
> Jonathan LaCour
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Robert Collins]

On 17 January 2014 06:39, Mark Washenberger
 wrote:
>

> There's a few more items here that are needed for glance to be able to work
> with requests (which we really really want).
> 1) Support for 100-expect-continue is probably going to be required in
> glance as well as swift

Is this currently supported? If not, frankly, I wouldn't bother. The
semantics in HTTP/2 are much better - I'd aim straight at that.

> 2) Support for turning off tls/ssl compression (our streams are already
> compressed)

Out of interest - whats the overhead of running tls compression
against compressed data? Is it really noticable?

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[John Utz]

Outlook Web MUA, forgive the top post. :-(

While a single import line that brings all the good stuff in at one shot is 
very convenient for the creation of an application, it would muddy the security 
picture *substantially* for the exact type of developer\customer that you would 
be targeting with this sort of syntactic sugar.

As Jesse alludes to below, the expanding tree of dependencies would be masked 
by the aggregation.

So, most likely, they would be pulling in vast numbers of things that they 
don't require to get their simple app done (there's an idea! an eclipse plugin 
that helpfully points out all the things that you are *not* using and offers to 
redo your imports for you :-) ).

As a result, when a security defect is published concerning one of those hidden 
dependencies, they will not have any reason to think that it effects them.

just my us$0.02;

johnu 

From: Jesse Noller [jesse.nol...@rackspace.com]
Sent: Thursday, January 16, 2014 5:42 PM
To: OpenStack Development Mailing List (not for usage questions)
Cc: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] a "common" client library

On Jan 16, 2014, at 4:59 PM, "Renat Akhmerov" 
mailto:rakhme...@mirantis.com>> wrote:

On 16 Jan 2014, at 13:06, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:

Since it’s pretty easy to get lost among all the opinions I’d like to 
clarify/ask a couple of things:


  *   Keeping all the clients physically separate/combining them in to a single 
library. Two things here:
 *   In case of combining them, what exact project are we considering? If 
this list is limited to core projects like nova and keystone what policy could 
we have for other projects to join this list? (Incubation, graduation, 
something else?)
 *   In terms of granularity and easiness of development I’m for keeping 
them separate but have them use the same boilerplate code, basically we need a 
OpenStack Rest Client Framework which is flexible enough to address all the 
needs in an abstract domain agnostic manner. I would assume that combining them 
would be an additional organizational burden that every stakeholder would have 
to deal with.

Keeping them separate is awesome for *us* but really, really, really sucks for 
users trying to use the system.

You may be right but not sure that adding another line into requirements.txt is 
a huge loss of usability.


It is when that 1 dependency pulls in 6 others that pull in 10 more - every 
little barrier or potential failure from the inability to make a static binary 
to how each tool acts different is a paper cut of frustration to an end user.

Most of the time the clients don't even properly install because of 
dependencies on setuptools plugins and other things. For developers (as I've 
said) the story is worse: you have potentially 22+ individual packages and 
their dependencies to deal with if they want to use a complete openstack 
install from their code.

So it doesn't boil down to just 1 dependency: it's a long laundry list of 
things that make consumers' lives more difficult and painful.

This doesn't even touch on the fact there aren't blessed SDKs or tools pointing 
users to consume openstack in their preferred programming language.

Shipping an API isn't enough - but it can be fixed easily enough.

Renat Akhmerov
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

On 17 Jan 2014, at 10:04, Jonathan LaCour  
wrote:

> "pip install openstack"


That would be awesome :)

Renat Akhmerov
@ Mirantis Inc.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jonathan LaCour]

On Thu, Jan 16, 2014 at 5:42 PM, Jesse Noller wrote:

>
>
> On Jan 16, 2014, at 4:59 PM, "Renat Akhmerov" 
> wrote:
>
>  On 16 Jan 2014, at 13:06, Jesse Noller 
> wrote:
>
>   Since it’s pretty easy to get lost among all the opinions I’d like to
> clarify/ask a couple of things:
>
>
>- Keeping all the clients physically separate/combining them in to a
>single library. Two things here:
>   - In case of combining them, what exact project are we considering?
>   If this list is limited to core projects like nova and keystone what 
> policy
>   could we have for other projects to join this list? (Incubation,
>   graduation, something else?)
>   - In terms of granularity and easiness of development I’m for
>   keeping them separate but have them use the same boilerplate code,
>   basically we need a OpenStack Rest Client Framework which is flexible
>   enough to address all the needs in an abstract domain agnostic manner. I
>   would assume that combining them would be an additional organizational
>   burden that every stakeholder would have to deal with.
>
>
>  Keeping them separate is awesome for *us* but really, really, really
> sucks for users trying to use the system.
>
>
> You may be right but not sure that adding another line into
> requirements.txt is a huge loss of usability.
>
>
>  It is when that 1 dependency pulls in 6 others that pull in 10 more -
> every little barrier or potential failure from the inability to make a
> static binary to how each tool acts different is a paper cut of frustration
> to an end user.
>

>
Most of the time the clients don't even properly install because of
> dependencies on setuptools plugins and other things. For developers (as
> I've said) the story is worse: you have potentially 22+ individual packages
> and their dependencies to deal with if they want to use a complete
> openstack install from their code.
>
>  So it doesn't boil down to just 1 dependency: it's a long laundry list
> of things that make consumers' lives more difficult and painful.
>
>  This doesn't even touch on the fact there aren't blessed SDKs or tools
> pointing users to consume openstack in their preferred programming language.
>
>  Shipping an API isn't enough - but it can be fixed easily enough.
>

+100

OpenStack must be attractive to our end users (developers and deployers),
as I mentioned earlier. Let's make it as simple as "pip install openstack"
if at all possible!

--
Jonathan LaCour
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jonathan LaCour]

On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft  wrote:

>
> On Jan 16, 2014, at 4:06 PM, Jesse Noller 
> wrote:
>
>
> On Jan 16, 2014, at 2:22 PM, Renat Akhmerov 
> wrote:
>
> Since it’s pretty easy to get lost among all the opinions I’d like to
> clarify/ask a couple of things:
>
>
>- Keeping all the clients physically separate/combining them in to a
>single library. Two things here:
>   - In case of combining them, what exact project are we considering?
>   If this list is limited to core projects like nova and keystone what 
> policy
>   could we have for other projects to join this list? (Incubation,
>   graduation, something else?)
>   - In terms of granularity and easiness of development I’m for
>   keeping them separate but have them use the same boilerplate code,
>   basically we need a OpenStack Rest Client Framework which is flexible
>   enough to address all the needs in an abstract domain agnostic manner. I
>   would assume that combining them would be an additional organizational
>   burden that every stakeholder would have to deal with.
>
>
> Keeping them separate is awesome for *us* but really, really, really sucks
> for users trying to use the system.
>
>
> I agree. Keeping them separate trades user usability for developer
> usability, I think user usability is a better thing to strive for.
>

100% agree with this. In order for OpenStack to be its most successful, I
believe firmly that a focus on end-users and deployers needs to take the
forefront. That means making OpenStack clouds as easy to consume/leverage
as possible for users and tool builders, and simplifying/streamlining as
much as possible.

I think that a single, common client project, based upon package
namespaces, with a unified, cohesive feel is a big step in this direction.

--
Jonathan LaCour
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Matthew Farina]

It seems we have two target audiences here. Developers who work on
OpenStack and developers who write apps to use it. In the long run I think
we need to optimize for both of these groups.

If we want developers to write applications to use OpenStack in python we
likely need a "common" python SDK.

Note, I'm not a fan of the term client because it's not the common language
for this group of developers.


On Fri, Jan 17, 2014 at 10:26 AM, John Dennis  wrote:

> >> Keeping them separate is awesome for *us* but really, really, really
> >> sucks for users trying to use the system.
> >
> > I agree. Keeping them separate trades user usability for developer
> > usability, I think user usability is a better thing to strive for.
>
> I don't understand how multiple independent code bases with a lot of
> overlapping code/logic is a win for developers. The more we can move to
> single shared code the easier code comprehension and maintenance
> becomes. From a software engineering perspective the amount of
> duplicated code/logic in OpenStack is worrisome. Iterating towards
> common code seems like a huge developer win as well as greatly enhancing
> robustness in the process.
>
> --
> John
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[John Dennis]

>> Keeping them separate is awesome for *us* but really, really, really
>> sucks for users trying to use the system. 
> 
> I agree. Keeping them separate trades user usability for developer
> usability, I think user usability is a better thing to strive for.

I don't understand how multiple independent code bases with a lot of
overlapping code/logic is a win for developers. The more we can move to
single shared code the easier code comprehension and maintenance
becomes. From a software engineering perspective the amount of
duplicated code/logic in OpenStack is worrisome. Iterating towards
common code seems like a huge developer win as well as greatly enhancing
robustness in the process.

-- 
John

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 8:36 PM, "Renat Akhmerov" 
mailto:rakhme...@mirantis.com>> wrote:

Ok, I think most of the reasoning you’ve said makes sense. So for a 
non-incubated project we’re going to have separate clients and then get them 
integrated into this single client once the project itself gets incubated? Or 
it’s going to happen when the project gets integrated into core os projects? So 
if yes, it’s just going to be one more incubation/integration requirement, 
right?


My gut says moving more code into Oslo and incurring that change for all 22(23 
with solum) client code based short term solves one aspect of cleaning up the 
clients - if - the sub projects agree that the code they are now depending on 
presents a useful API to them. It's not clear to me that case has been 
explained, or the design of those new libraries sussed out.

My personal opinion is that if people wanted the current consolidation to 
continue we let that go, but begin fleshing out what a stand alone 
project/library akin to horizon would look like (let's call it zenith).

If we go this route we need to buildout some blueprints with design / layout / 
APIs for people to review and approve.

When talking with dean today we actually see it as fewer layers but the most 
work : benefit comes from unifying and coordinating the client code from each 
of the projects. This latter part requires consensus from the teams on the 
layout and design, contributing to it, etc

Dean and I started an etherpad today, I need to go back through this thread and 
capture more design considerations and properly capture it in the wiki.

With at least a fleshed out strawman proposal for layout and design we can 
discuss concrete things. I think some of the contention is focused on the 
currently progressing work on a blueprint that doesn't fully capture the design 
and I'd like to avoid that.

Renat

On 16 Jan 2014, at 18:09, Donald Stufft 
mailto:don...@stufft.io>> wrote:


On Jan 16, 2014, at 8:42 PM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:



On Jan 16, 2014, at 4:59 PM, "Renat Akhmerov" 
mailto:rakhme...@mirantis.com>> wrote:

On 16 Jan 2014, at 13:06, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:

Since it’s pretty easy to get lost among all the opinions I’d like to 
clarify/ask a couple of things:


  *   Keeping all the clients physically separate/combining them in to a single 
library. Two things here:
 *   In case of combining them, what exact project are we considering? If 
this list is limited to core projects like nova and keystone what policy could 
we have for other projects to join this list? (Incubation, graduation, 
something else?)
 *   In terms of granularity and easiness of development I’m for keeping 
them separate but have them use the same boilerplate code, basically we need a 
OpenStack Rest Client Framework which is flexible enough to address all the 
needs in an abstract domain agnostic manner. I would assume that combining them 
would be an additional organizational burden that every stakeholder would have 
to deal with.

Keeping them separate is awesome for *us* but really, really, really sucks for 
users trying to use the system.

You may be right but not sure that adding another line into requirements.txt is 
a huge loss of usability.


It is when that 1 dependency pulls in 6 others that pull in 10 more - every 
little barrier or potential failure from the inability to make a static binary 
to how each tool acts different is a paper cut of frustration to an end user.

Most of the time the clients don't even properly install because of 
dependencies on setuptools plugins and other things. For developers (as I've 
said) the story is worse: you have potentially 22+ individual packages and 
their dependencies to deal with if they want to use a complete openstack 
install from their code.

So it doesn't boil down to just 1 dependency: it's a long laundry list of 
things that make consumers' lives more difficult and painful.

This doesn't even touch on the fact there aren't blessed SDKs or tools pointing 
users to consume openstack in their preferred programming language.

Shipping an API isn't enough - but it can be fixed easily enough.

There’s also the discovery problem, it’s incredibly frustrating if, as I’m 
starting out to use an Openstack based cloud, everytime I want to touch some 
new segment of the service I need to go find out what the client lib is for 
that, possibly download the dependencies for it, possibly get it approved, etc.

Splitting up services makes a lot of sense on the server side, but to the 
consumer a cloud often times isn’t a disjoint set of services that happen to be 
working in parallel, it is a single unified product where they may not know the 
boundary lines, or at the very least the boundaries can be fuzzy for them.


Renat Akhmerov
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org

Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

Ok, I think most of the reasoning you’ve said makes sense. So for a 
non-incubated project we’re going to have separate clients and then get them 
integrated into this single client once the project itself gets incubated? Or 
it’s going to happen when the project gets integrated into core os projects? So 
if yes, it’s just going to be one more incubation/integration requirement, 
right?

Renat

On 16 Jan 2014, at 18:09, Donald Stufft  wrote:

> 
> On Jan 16, 2014, at 8:42 PM, Jesse Noller  wrote:
> 
>> 
>> 
>> On Jan 16, 2014, at 4:59 PM, "Renat Akhmerov"  wrote:
>> 
>>> On 16 Jan 2014, at 13:06, Jesse Noller  wrote:
>>> 
> Since it’s pretty easy to get lost among all the opinions I’d like to 
> clarify/ask a couple of things:
> 
> Keeping all the clients physically separate/combining them in to a single 
> library. Two things here:
> In case of combining them, what exact project are we considering? If this 
> list is limited to core projects like nova and keystone what policy could 
> we have for other projects to join this list? (Incubation, graduation, 
> something else?)
> In terms of granularity and easiness of development I’m for keeping them 
> separate but have them use the same boilerplate code, basically we need a 
> OpenStack Rest Client Framework which is flexible enough to address all 
> the needs in an abstract domain agnostic manner. I would assume that 
> combining them would be an additional organizational burden that every 
> stakeholder would have to deal with.
 
 Keeping them separate is awesome for *us* but really, really, really sucks 
 for users trying to use the system. 
>>> 
>>> You may be right but not sure that adding another line into 
>>> requirements.txt is a huge loss of usability.
>>> 
>> 
>> It is when that 1 dependency pulls in 6 others that pull in 10 more - every 
>> little barrier or potential failure from the inability to make a static 
>> binary to how each tool acts different is a paper cut of frustration to an 
>> end user.
>> 
>> Most of the time the clients don't even properly install because of 
>> dependencies on setuptools plugins and other things. For developers (as I've 
>> said) the story is worse: you have potentially 22+ individual packages and 
>> their dependencies to deal with if they want to use a complete openstack 
>> install from their code.
>> 
>> So it doesn't boil down to just 1 dependency: it's a long laundry list of 
>> things that make consumers' lives more difficult and painful.
>> 
>> This doesn't even touch on the fact there aren't blessed SDKs or tools 
>> pointing users to consume openstack in their preferred programming language.
>> 
>> Shipping an API isn't enough - but it can be fixed easily enough.
> 
> There’s also the discovery problem, it’s incredibly frustrating if, as I’m 
> starting out to use an Openstack based cloud, everytime I want to touch some 
> new segment of the service I need to go find out what the client lib is for 
> that, possibly download the dependencies for it, possibly get it approved, 
> etc. 
> 
> Splitting up services makes a lot of sense on the server side, but to the 
> consumer a cloud often times isn’t a disjoint set of services that happen to 
> be working in parallel, it is a single unified product where they may not 
> know the boundary lines, or at the very least the boundaries can be fuzzy for 
> them.
> 
>> 
>>> Renat Akhmerov
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> -
> Donald Stufft
> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Donald Stufft]

On Jan 16, 2014, at 8:42 PM, Jesse Noller  wrote:

> 
> 
> On Jan 16, 2014, at 4:59 PM, "Renat Akhmerov"  wrote:
> 
>> On 16 Jan 2014, at 13:06, Jesse Noller  wrote:
>> 
 Since it’s pretty easy to get lost among all the opinions I’d like to 
 clarify/ask a couple of things:
 
 Keeping all the clients physically separate/combining them in to a single 
 library. Two things here:
 In case of combining them, what exact project are we considering? If this 
 list is limited to core projects like nova and keystone what policy could 
 we have for other projects to join this list? (Incubation, graduation, 
 something else?)
 In terms of granularity and easiness of development I’m for keeping them 
 separate but have them use the same boilerplate code, basically we need a 
 OpenStack Rest Client Framework which is flexible enough to address all 
 the needs in an abstract domain agnostic manner. I would assume that 
 combining them would be an additional organizational burden that every 
 stakeholder would have to deal with.
>>> 
>>> Keeping them separate is awesome for *us* but really, really, really sucks 
>>> for users trying to use the system. 
>> 
>> You may be right but not sure that adding another line into requirements.txt 
>> is a huge loss of usability.
>> 
> 
> It is when that 1 dependency pulls in 6 others that pull in 10 more - every 
> little barrier or potential failure from the inability to make a static 
> binary to how each tool acts different is a paper cut of frustration to an 
> end user.
> 
> Most of the time the clients don't even properly install because of 
> dependencies on setuptools plugins and other things. For developers (as I've 
> said) the story is worse: you have potentially 22+ individual packages and 
> their dependencies to deal with if they want to use a complete openstack 
> install from their code.
> 
> So it doesn't boil down to just 1 dependency: it's a long laundry list of 
> things that make consumers' lives more difficult and painful.
> 
> This doesn't even touch on the fact there aren't blessed SDKs or tools 
> pointing users to consume openstack in their preferred programming language.
> 
> Shipping an API isn't enough - but it can be fixed easily enough.

There’s also the discovery problem, it’s incredibly frustrating if, as I’m 
starting out to use an Openstack based cloud, everytime I want to touch some 
new segment of the service I need to go find out what the client lib is for 
that, possibly download the dependencies for it, possibly get it approved, etc. 

Splitting up services makes a lot of sense on the server side, but to the 
consumer a cloud often times isn’t a disjoint set of services that happen to be 
working in parallel, it is a single unified product where they may not know the 
boundary lines, or at the very least the boundaries can be fuzzy for them.

> 
>> Renat Akhmerov
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 4:59 PM, "Renat Akhmerov" 
mailto:rakhme...@mirantis.com>> wrote:

On 16 Jan 2014, at 13:06, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:

Since it’s pretty easy to get lost among all the opinions I’d like to 
clarify/ask a couple of things:


  *   Keeping all the clients physically separate/combining them in to a single 
library. Two things here:
 *   In case of combining them, what exact project are we considering? If 
this list is limited to core projects like nova and keystone what policy could 
we have for other projects to join this list? (Incubation, graduation, 
something else?)
 *   In terms of granularity and easiness of development I’m for keeping 
them separate but have them use the same boilerplate code, basically we need a 
OpenStack Rest Client Framework which is flexible enough to address all the 
needs in an abstract domain agnostic manner. I would assume that combining them 
would be an additional organizational burden that every stakeholder would have 
to deal with.

Keeping them separate is awesome for *us* but really, really, really sucks for 
users trying to use the system.

You may be right but not sure that adding another line into requirements.txt is 
a huge loss of usability.


It is when that 1 dependency pulls in 6 others that pull in 10 more - every 
little barrier or potential failure from the inability to make a static binary 
to how each tool acts different is a paper cut of frustration to an end user.

Most of the time the clients don't even properly install because of 
dependencies on setuptools plugins and other things. For developers (as I've 
said) the story is worse: you have potentially 22+ individual packages and 
their dependencies to deal with if they want to use a complete openstack 
install from their code.

So it doesn't boil down to just 1 dependency: it's a long laundry list of 
things that make consumers' lives more difficult and painful.

This doesn't even touch on the fact there aren't blessed SDKs or tools pointing 
users to consume openstack in their preferred programming language.

Shipping an API isn't enough - but it can be fixed easily enough.

Renat Akhmerov
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jamie Lennox]

*snip*
> 
> 
> Distinction between client layers won't get lost and would only be
> improved. My basic idea is the following:
> 
> 1) Transport layer would handle all transport related stuff - HTTP,
> JSON encoding, auth, caching, etc.
> 
> 2) Model layer (Resource classes, BaseManager, etc.) will handle data
> representation, validation
> 
> 3) API layer will handle all project specific stuff - url mapping,
> etc. (This will be imported to use client in other applications)
> 
> 4) Cli level will handle all stuff related to cli mapping - argparse,
> argcomplete, etc.
> 
> 
> >I believe the current effort referenced by the blueprint is focusing
> on moving existing code into the incubator for reuse, to make it
> easier to restructure later. Alexei, do I have that correct?
> 
> You are right. The first thing we do is try to make all clients
> look/work in similar way. After we'll continue our work with improving
> overall structure.

Wow, this thread took off. I'll jump in here as it's as good a point as
any. Sorry this will be long but i've been working on this for a while. 

So hopefully a few people know that we have been looking at addressing
this problem in keystoneclient as it really needs to be a/the base
client as it will handle authentication across all the projects.

I completely agree with your 4 sections, though i would really love to
see some more detail around levels 2 and 3. 

Section 1: Transport layer

For the majority i'm hoping that section 1 is in progress to being
solved. Keystoneclient introduced a session object which is designed to
handle HTTP communication for all the clients. This is a reusable object
and will contain an auth plugin such that all communication through the
session object will be authenticated. There is follow on work here to
have the session object manage the service catalog for clients so that a
client will simply specify the service type and version it needs to
communicate with. You can see the work around auth plugins and the
follow up here: https://review.openstack.org/#/c/60751/ (be aware that
backwards compatibility issues make it an ugly patch).

This solution was discussed at the HK summit and was generally accepted.
For an earlier draft of the vision around this see:
http://www.jamielennox.net/blog/2013/09/27/apiclient-communications/
which i should post a revision of. 

With this is mind a base client should look like not much more than:
https://review.openstack.org/#/c/60751/5/keystoneclient/baseclient.py


Section 4: CLI

Regarding section 4, my understanding is that all clients are moving
towards OSC for there CLI component. This is great. I personally like
the idea of the various python-*clients being controlled by their own
project as they are the people who best understand how to interact with
their services and design an API that makes sense to them.

I then would prefer that the CLI is controlled completely by OSC (not
plugins back to the client) because this way we end up with one
centralized choke point so that the CLI interface is common across all
the projects. Client developers will need to submit reviews to OSC to
add new CLI commands and connect it back to the client library. This
however is up to Dean and OSC, but i would prefer them to keep an eye on
this interface over the current system.


Section 2 & 3: Models and Managers

This is the part i've generally been concerned about. Firstly the idea
of having client code in OSLO seems stupid to me. Unlike the servers the
clients will often live in the same process as each other, particularly
with regards to exceptions this means that every client library now has
the same list of exceptions though in their own namespace so there is no
common way to handle an exception type when dealing with multiple
clients.

My concern with the common client code in OSLO is specifically that at
no time has there ever been a conversation about what a client should be
trying to achieve or the best way to do it. The code in apiclient is a
snapshot of what was in use across a number of projects at that time and
NOT any attempt to define a best practice. Whilst I understand the
desire to standardize, I and keystoneclient take backwards compatibility
very seriously and there is a lot of code in apiclient which does not
fit what I think a client should do and i do not want to support it
going forward.

Having said that I am very interested in figuring this out and defining
a common client framework. But it seems to me the sort of conversation
that we want to have at a summit session. Building this common framework
based on the current state of the clients is just formalizing the
current disaster.



To summarize, great! I'm the number one supported of improving and
unifying the clients. Please under there is existing work underway
towards this goal and that blindly copying oslo's apiclient everywhere
may be good for standardization but I don't think it really improves
anything and i'm not convinced it will make it ea

Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

On 16 Jan 2014, at 13:06, Jesse Noller  wrote:

>> Since it’s pretty easy to get lost among all the opinions I’d like to 
>> clarify/ask a couple of things:
>> 
>> Keeping all the clients physically separate/combining them in to a single 
>> library. Two things here:
>> In case of combining them, what exact project are we considering? If this 
>> list is limited to core projects like nova and keystone what policy could we 
>> have for other projects to join this list? (Incubation, graduation, 
>> something else?)
>> In terms of granularity and easiness of development I’m for keeping them 
>> separate but have them use the same boilerplate code, basically we need a 
>> OpenStack Rest Client Framework which is flexible enough to address all the 
>> needs in an abstract domain agnostic manner. I would assume that combining 
>> them would be an additional organizational burden that every stakeholder 
>> would have to deal with.
> 
> Keeping them separate is awesome for *us* but really, really, really sucks 
> for users trying to use the system. 

You may be right but not sure that adding another line into requirements.txt is 
a huge loss of usability.

Renat Akhmerov___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

On 16 Jan 2014, at 12:36, Dean Troyer  wrote:

> I've already written a POC for solum and some other things to demonstrate how 
> to add additional projects simply by installing the python-*client package.  
> https://github.com/dtroyer/python-oscplugin is a trivial example.

Thanks, this link is helpful.
> In terms of granularity and easiness of development I’m for keeping them 
> separate but have them use the same boilerplate code, basically we need a 
> OpenStack Rest Client Framework which is flexible enough to address all the 
> needs in an abstract domain agnostic manner. I would assume that combining 
> them would be an additional organizational burden that every stakeholder 
> would have to deal with.
> If it is not a library that is actually shared you will get back to the 
> current situation over time. 

How is that different from any other oslo stuff?
> Has anyone ever considered an idea of generating a fully functional REST 
> client automatically based on an API specification (WADL could be used for 
> that)? Not sure how convenient it would be, it really depends on a particular 
> implementation, but as an idea it could be at least thought of. Sounds a 
> little bit crazy though, I recognize it :).
> 
> When you have stable and accurate documents of this sort, let's talk.  You 
> may have noticed that few (any?) of the recent major API revs are documented 
> in this manner…

There’s still something to work on for us.. As far as what I’ve written, just a 
crazy idea that came to my mind. After all, everyone’s phone has lots of things 
that were considered crazy years ago.

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Donald Stufft]

On Jan 16, 2014, at 3:22 PM, Renat Akhmerov  wrote:

> Has anyone ever considered an idea of generating a fully functional REST 
> client automatically based on an API specification (WADL could be used for 
> that)? Not sure how convenient it would be, it really depends on a particular 
> implementation, but as an idea it could be at least thought of. Sounds a 
> little bit crazy though, I recognize it :).


Also please no. If you want “automatic client” stop being “REST” (which has 
been diluted to be meaningless or at least nothing like it’s actual definition) 
and be REST and/or Hypermedia. Things like WADL are completely antagonistic to 
actual REST (aka Hypermedia).

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Donald Stufft]

On Jan 16, 2014, at 4:06 PM, Jesse Noller  wrote:

> 
> On Jan 16, 2014, at 2:22 PM, Renat Akhmerov  wrote:
> 
>> Since it’s pretty easy to get lost among all the opinions I’d like to 
>> clarify/ask a couple of things:
>> 
>> Keeping all the clients physically separate/combining them in to a single 
>> library. Two things here:
>> In case of combining them, what exact project are we considering? If this 
>> list is limited to core projects like nova and keystone what policy could we 
>> have for other projects to join this list? (Incubation, graduation, 
>> something else?)
>> In terms of granularity and easiness of development I’m for keeping them 
>> separate but have them use the same boilerplate code, basically we need a 
>> OpenStack Rest Client Framework which is flexible enough to address all the 
>> needs in an abstract domain agnostic manner. I would assume that combining 
>> them would be an additional organizational burden that every stakeholder 
>> would have to deal with.
> 
> Keeping them separate is awesome for *us* but really, really, really sucks 
> for users trying to use the system. 

I agree. Keeping them separate trades user usability for developer usability, I 
think user usability is a better thing to strive for.


> 
>> Has anyone ever considered an idea of generating a fully functional REST 
>> client automatically based on an API specification (WADL could be used for 
>> that)? Not sure how convenient it would be, it really depends on a 
>> particular implementation, but as an idea it could be at least thought of. 
>> Sounds a little bit crazy though, I recognize it :).
>> 
>> Renat Akhmerov
>> 
>> On 16 Jan 2014, at 11:52, Chmouel Boudjnah  wrote:
>> 
>>> On Thu, Jan 16, 2014 at 8:40 PM, Donald Stufft  wrote:
>>> 
>>> On Jan 16, 2014, at 2:36 PM, Joe Gordon  wrote:
>>> 
 2) major overhaul of client libraries so they are all based off a common 
 base library. This would cover namespace changes, and possible a push to 
 move CLI into python-openstackclient
>>> This seems like the biggest win to me. 
>>> 
>>> 
>>> +1 
>>> 
>>> Chmouel. 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 2:22 PM, Renat Akhmerov 
mailto:rakhme...@mirantis.com>> wrote:

Since it’s pretty easy to get lost among all the opinions I’d like to 
clarify/ask a couple of things:


  *   Keeping all the clients physically separate/combining them in to a single 
library. Two things here:
 *   In case of combining them, what exact project are we considering? If 
this list is limited to core projects like nova and keystone what policy could 
we have for other projects to join this list? (Incubation, graduation, 
something else?)
 *   In terms of granularity and easiness of development I’m for keeping 
them separate but have them use the same boilerplate code, basically we need a 
OpenStack Rest Client Framework which is flexible enough to address all the 
needs in an abstract domain agnostic manner. I would assume that combining them 
would be an additional organizational burden that every stakeholder would have 
to deal with.

Keeping them separate is awesome for *us* but really, really, really sucks for 
users trying to use the system.


  *   Has anyone ever considered an idea of generating a fully functional REST 
client automatically based on an API specification (WADL could be used for 
that)? Not sure how convenient it would be, it really depends on a particular 
implementation, but as an idea it could be at least thought of. Sounds a little 
bit crazy though, I recognize it :).

Renat Akhmerov

On 16 Jan 2014, at 11:52, Chmouel Boudjnah 
mailto:chmo...@enovance.com>> wrote:

On Thu, Jan 16, 2014 at 8:40 PM, Donald Stufft 
mailto:don...@stufft.io>> wrote:

On Jan 16, 2014, at 2:36 PM, Joe Gordon 
mailto:joe.gord...@gmail.com>> wrote:

2) major overhaul of client libraries so they are all based off a common base 
library. This would cover namespace changes, and possible a push to move CLI 
into python-openstackclient
This seems like the biggest win to me.


+1

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Dean Troyer]

On Thu, Jan 16, 2014 at 2:22 PM, Renat Akhmerov wrote:

>
>
>- Keeping all the clients physically separate/combining them in to a
>single library. Two things here:
>   - In case of combining them, what exact project are we considering?
>   If this list is limited to core projects like nova and keystone what 
> policy
>   could we have for other projects to join this list? (Incubation,
>   graduation, something else?)
>
>  FWIW, OpenStackClient has Identity built-in as very little else is useful
without it.  Compute, Image and Volume are in the repo but structured using
the new plugin system to make sure that works well.  Object-store is in the
repo but I consider it experimental as it also includes the project API
layer and my first cut at a common REST client.

I've already written a POC for solum and some other things to demonstrate
how to add additional projects simply by installing the python-*client
package.  https://github.com/dtroyer/python-oscplugin is a trivial example.

>
>- In terms of granularity and easiness of development I’m for keeping
>   them separate but have them use the same boilerplate code, basically we
>   need a OpenStack Rest Client Framework which is flexible enough to 
> address
>   all the needs in an abstract domain agnostic manner. I would assume that
>   combining them would be an additional organizational burden that every
>   stakeholder would have to deal with.
>
> If it is not a library that is actually shared you will get back to the
current situation over time.

>
>- Has anyone ever considered an idea of generating a fully functional
>REST client automatically based on an API specification (WADL could be used
>for that)? Not sure how convenient it would be, it really depends on a
>particular implementation, but as an idea it could be at least thought of.
>Sounds a little bit crazy though, I recognize it :).
>
> When you have stable and accurate documents of this sort, let's talk.  You
may have noticed that few (any?) of the recent major API revs are
documented in this manner...

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

Since it’s pretty easy to get lost among all the opinions I’d like to 
clarify/ask a couple of things:

Keeping all the clients physically separate/combining them in to a single 
library. Two things here:
In case of combining them, what exact project are we considering? If this list 
is limited to core projects like nova and keystone what policy could we have 
for other projects to join this list? (Incubation, graduation, something else?)
In terms of granularity and easiness of development I’m for keeping them 
separate but have them use the same boilerplate code, basically we need a 
OpenStack Rest Client Framework which is flexible enough to address all the 
needs in an abstract domain agnostic manner. I would assume that combining them 
would be an additional organizational burden that every stakeholder would have 
to deal with.
Has anyone ever considered an idea of generating a fully functional REST client 
automatically based on an API specification (WADL could be used for that)? Not 
sure how convenient it would be, it really depends on a particular 
implementation, but as an idea it could be at least thought of. Sounds a little 
bit crazy though, I recognize it :).

Renat Akhmerov

On 16 Jan 2014, at 11:52, Chmouel Boudjnah  wrote:

> On Thu, Jan 16, 2014 at 8:40 PM, Donald Stufft  wrote:
> 
> On Jan 16, 2014, at 2:36 PM, Joe Gordon  wrote:
> 
>> 2) major overhaul of client libraries so they are all based off a common 
>> base library. This would cover namespace changes, and possible a push to 
>> move CLI into python-openstackclient
> This seems like the biggest win to me. 
> 
> 
> +1 
> 
> Chmouel. 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Chmouel Boudjnah]

On Thu, Jan 16, 2014 at 8:40 PM, Donald Stufft  wrote:

>
> On Jan 16, 2014, at 2:36 PM, Joe Gordon  wrote:
>
> 2) major overhaul of client libraries so they are all based off a common
> base library. This would cover namespace changes, and possible a push to
> move CLI into python-openstackclient
>
> This seems like the biggest win to me.
>
>
+1

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Douglas Mendizabal]

+1 to a stand alone library for this.

>> 1) We would have to maintain rationale versioning and backwards compatibility
of this library. If we start library from scratch we'll have to add/change lots
of stuff before we'll reach some stability period.

I don’t think this is a hard problem to solve.  I think something like
semantic versioning fits the bill, and allows for backwards-incompatible
changes.  

>> 2)Another option would be to follow waterfall process and create a solid
library interface before including it to all client projects. However such
approach this period can take unknown amount of time and can be easily failed
during integration stage cause requirements change or some other reason.

Again, I think semver could address this.  Iterate over 0.x.x series, and
projects that don’t want to deal with constantly-changing code wouldn’t need
to jump on board until a 1.0.0 release.

-Doug M.

From:  Alexei Kornienko 
Reply-To:  "OpenStack Development Mailing List (not for usage questions)"

Date:  Thursday, January 16, 2014 at 1:03 PM
To:  "openstack-dev@lists.openstack.org" 
Subject:  Re: [openstack-dev] a "common" client library

Hello Joe,

> continuous refactoring and syncing across 22+ repositories sounds like a
> nightmare, one that I would like to avoid.
You are right this is not easy.

However I have several reasons to do that:
The hardest part is to bring basic stuff in sync across all projects (That's
what we are doing now). Later we'll work directly with oslo lib and just
sync changes from it.

We could introduce a standalone library to avoid the need to sync oslo code
across all projects but it brings additional problems:

1) We would have to maintain rationale versioning and backwards
compatibility of this library. If we start library from scratch we'll have
to add/change lots of stuff before we'll reach some stability period.

2)Another option would be to follow waterfall process and create a solid
library interface before including it to all client projects. However such
approach this period can take unknown amount of time and can be easily
failed during integration stage cause requirements change or some other
reason.

Please let me know what you think.

Best Regards,
Alexei

On 01/16/2014 08:16 PM, Joe Gordon wrote:
> 
> 
> 
> On Thu, Jan 16, 2014 at 12:07 PM, Alexei Kornienko
>  wrote:
>> On 01/16/2014 06:15 PM, Jesse Noller wrote:
>>> 
>>> On Jan 16, 2014, at 9:54 AM, Alexei Kornienko 
>>> wrote:
>>> 
>>>> On 01/16/2014 05:25 PM, Jesse Noller wrote:
>>>>> 
>>>>> On Jan 16, 2014, at 9:07 AM, Joe Gordon  wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller
>>>>>>  wrote:
>>>>>>> 
>>>>>>> On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones  wrote:
>>>>>>> Once a common library is in place, is there any intention to (or
>>>>>>> resistance against) collapsing the clients into a single project or even
>>>>>>> a single command (a la busybox)?
>>>>>>> 
>>>>>>> 
>>>>>>> that's what openstackclient is here for
>>>>>>> https://github.com/openstack/python-openstackclient
>>>>>>> 
>>>>>>> After speaking with people working on OSC and looking at the code base
>>>>>>> in depth; I don’t think this addresses what Chris is implying: OSC wraps
>>>>>>> the individual CLIs built by each project today, instead of the inverse:
>>>>>>> a common backend that the individual CLIs can wrap - the latter is an
>>>>>>> important distinction as currently, building a single binary install of
>>>>>>> OSC for say, Windows is difficult given the dependency tree incurred by
>>>>>>> each of the wrapped CLIs, difference in dependencies, structure, etc.
>>>>>>> 
>>>>>>> Also, wrapping a series of inconsistent back end Client classes /
>>>>>>> functions / methods means that the layer that presents a consistent user
>>>>>>> interface (OSC) to the user is made more complex juggling
>>>>>>> names/renames/commands/etc.
>>>>>>> 
>>>>>>> In the inverted case of what we have today (single backend); as a
>>>>>>> developer of user i

Re: [openstack-dev] a "common" client library

[Donald Stufft]

On Jan 16, 2014, at 2:36 PM, Joe Gordon  wrote:

> 2) major overhaul of client libraries so they are all based off a common base 
> library. This would cover namespace changes, and possible a push to move CLI 
> into python-openstackclient


This seems like the biggest win to me. 

-
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Joe Gordon]

On Thu, Jan 16, 2014 at 2:03 PM, Alexei Kornienko <
alexei.kornie...@gmail.com> wrote:

>  Hello Joe,
>
>
> continuous refactoring and syncing across 22+ repositories sounds like a
> nightmare, one that I would like to avoid.
>
> You are right this is not easy.
>
> However I have several reasons to do that:
> The hardest part is to bring basic stuff in sync across all projects
> (That's what we are doing now). Later we'll work directly with oslo lib and
> just sync changes from it.
>

My concern here is that is just extra churn for minimal value, assuming
there are plans to completely overhaul this in the near future.

Perhaps this blueprint is really two:

1) do short term work to reduplicate the common client functionality. This
would cover the current patches and involve oslo-incubator, and is limited
in scope and shouldn't require to many oslo sync iterations.

2) major overhaul of client libraries so they are all based off a common
base library. This would cover namespace changes, and possible a push to
move CLI into python-openstackclient


> We could introduce a standalone library to avoid the need to sync oslo
> code across all projects but it brings additional problems:
>
> 1) We would have to maintain rationale versioning and backwards
> compatibility of this library. If we start library from scratch we'll have
> to add/change lots of stuff before we'll reach some stability period.
>
> 2)Another option would be to follow waterfall process and create a solid
> library interface before including it to all client projects. However such
> approach this period can take unknown amount of time and can be easily
> failed during integration stage cause requirements change or some other
> reason.
>
> Please let me know what you think.
>
> Best Regards,
> Alexei
>
>
> On 01/16/2014 08:16 PM, Joe Gordon wrote:
>
>
>
>
> On Thu, Jan 16, 2014 at 12:07 PM, Alexei Kornienko <
> alexei.kornie...@gmail.com> wrote:
>
>>   On 01/16/2014 06:15 PM, Jesse Noller wrote:
>>
>>
>>  On Jan 16, 2014, at 9:54 AM, Alexei Kornienko <
>> alexei.kornie...@gmail.com> wrote:
>>
>>  On 01/16/2014 05:25 PM, Jesse Noller wrote:
>>
>>
>>  On Jan 16, 2014, at 9:07 AM, Joe Gordon  wrote:
>>
>>
>>
>>
>> On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller > > wrote:
>>
>>>
>>>   On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah 
>>> wrote:
>>>
>>>
>>> On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones  wrote:
>>>
 Once a common library is in place, is there any intention to (or
 resistance against) collapsing the clients into a single project or even a
 single command (a la busybox)?
>>>
>>>
>>>
>>>  that's what openstackclient is here for
>>> https://github.com/openstack/python-openstackclient
>>>
>>>
>>>   After speaking with people working on OSC and looking at the code
>>> base in depth; I don’t think this addresses what Chris is implying: OSC
>>> wraps the individual CLIs built by each project today, instead of the
>>> inverse: a common backend that the individual CLIs can wrap - the latter is
>>> an important distinction as currently, building a single binary install of
>>> OSC for say, Windows is difficult given the dependency tree incurred by
>>> each of the wrapped CLIs, difference in dependencies, structure, etc.
>>>
>>>  Also, wrapping a series of inconsistent back end Client classes /
>>> functions / methods means that the layer that presents a consistent user
>>> interface (OSC) to the user is made more complex juggling
>>> names/renames/commands/etc.
>>>
>>>  In the inverted case of what we have today (single backend); as a
>>> developer of user interfaces (CLIs, Applications, Web apps (horizon)) you
>>> would be able to:
>>>
>>>  from openstack.common.api import Auth
>>>  from openstack.common.api import Compute
>>> from openstack.common.util import cli_tools
>>>
>>>  my_cli = cli_tools.build(…)
>>>
>>>  def my_command(cli):
>>> compute = Compute(Auth(cli.tentant…, connect=True))
>>> compute.list_flavors()
>>>
>>>  This would mean that *even if the individual clients needed or wanted
>>> to keep their specific CLIs, they would be able to use a not “least common
>>> denominator” back end (each service can have a rich
>>> common.api.compute.py or api.compute/client.py and extend where needed.
>>> However tools like horizon / openstackclient can choose not to leverage the
>>> “power user/operator/admin” components and present a simplified user
>>> interface.
>>>
>>>  I’m working on a wiki page + blueprint to brainstorm how we could
>>> accomplish this based off of what work is in flight today (see doug’s
>>> linked blueprint) and sussing out a layout / API strawman for discussion.
>>>
>>>  Some of the additions that came out of this email threads and others:
>>>
>>>  1. Common backend should provide / offer caching utilities
>>> 2. Auth retries need to be handled by the auth object, and each
>>> sub-project delegates to the auth object to manage that.
>>> 3. Verified Mocks / Stubs / Fakes must be provided for proper unit
>>> test

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

Hello Joe,

continuous refactoring and syncing across 22+ repositories sounds like 
a nightmare, one that I would like to avoid.

You are right this is not easy.

However I have several reasons to do that:
The hardest part is to bring basic stuff in sync across all projects 
(That's what we are doing now). Later we'll work directly with oslo lib 
and just sync changes from it.


We could introduce a standalone library to avoid the need to sync oslo 
code across all projects but it brings additional problems:


1) We would have to maintain rationale versioning and backwards 
compatibility of this library. If we start library from scratch we'll 
have to add/change lots of stuff before we'll reach some stability period.


2)Another option would be to follow waterfall process and create a solid 
library interface before including it to all client projects. However 
such approach this period can take unknown amount of time and can be 
easily failed during integration stage cause requirements change or some 
other reason.


Please let me know what you think.

Best Regards,
Alexei

On 01/16/2014 08:16 PM, Joe Gordon wrote:




On Thu, Jan 16, 2014 at 12:07 PM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:


On 01/16/2014 06:15 PM, Jesse Noller wrote:


On Jan 16, 2014, at 9:54 AM, Alexei Kornienko
mailto:alexei.kornie...@gmail.com>>
wrote:


On 01/16/2014 05:25 PM, Jesse Noller wrote:


On Jan 16, 2014, at 9:07 AM, Joe Gordon mailto:joe.gord...@gmail.com>> wrote:





On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller
mailto:jesse.nol...@rackspace.com>> wrote:


On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah
mailto:chmo...@enovance.com>> wrote:



On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones
mailto:c...@tenshu.net>> wrote:

Once a common library is in place, is there any
intention to (or resistance against) collapsing the
clients into a single project or even a single
command (a la busybox)?



that's what openstackclient is here for
https://github.com/openstack/python-openstackclient


After speaking with people working on OSC and looking at
the code base in depth; I don't think this addresses what
Chris is implying: OSC wraps the individual CLIs built by
each project today, instead of the inverse: a common
backend that the individual CLIs can wrap - the latter is
an important distinction as currently, building a single
binary install of OSC for say, Windows is difficult given
the dependency tree incurred by each of the wrapped CLIs,
difference in dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client
classes / functions / methods means that the layer that
presents a consistent user interface (OSC) to the user is
made more complex juggling names/renames/commands/etc.

In the inverted case of what we have today (single
backend); as a developer of user interfaces (CLIs,
Applications, Web apps (horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(...)

def my_command(cli):
compute = Compute(Auth(cli.tentant..., connect=True))
compute.list_flavors()

This would mean that *even if the individual clients
needed or wanted to keep their specific CLIs, they would
be able to use a not "least common denominator" back end
(each service can have a rich common.api.compute.py
 or api.compute/client.py
and extend where needed. However tools like horizon /
openstackclient can choose not to leverage the "power
user/operator/admin" components and present a simplified
user interface.

I'm working on a wiki page + blueprint to brainstorm how
we could accomplish this based off of what work is in
flight today (see doug's linked blueprint) and sussing out
a layout / API strawman for discussion.

Some of the additions that came out of this email threads
and others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and
each sub-project delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for
proper unit testing


I am happy to see this work being done, there is definitely a
lot of work to be done on the clients.

This blueprint sounds like its still being fleshed out, so I
am wondering what the value is of the current patches
https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z

Those patches mainly sync cliutils and apiutils fro

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 11:39 AM, Mark Washenberger 
mailto:mark.washenber...@markwash.net>> wrote:




On Thu, Jan 16, 2014 at 8:06 AM, Dean Troyer 
mailto:dtro...@gmail.com>> wrote:
On Thu, Jan 16, 2014 at 9:37 AM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:
On Jan 16, 2014, at 9:26 AM, Justin Hammond 
mailto:justin.hamm...@rackspace.com>> wrote:
I'm not sure if it was said, but which httplib using being used (urllib3
maybe?). Also I noticed many people were talking about supporting auth
properly, but are there any intentions to properly support 'noauth'
(python-neutronclient, for instance, doesn't support it properly as of
this writing)?
Can you detail out noauth for me; and I would say the defacto httplib in python 
today is python-requests - urllib3 is also good but I would say from a 
*consumer* standpoint requests offers more in terms of usability / extensibility

requests is built on top of urllib3 so there's that...

The biggest reaon I favor using Jamie Lennox's new session layer stuff in 
keystoneclient is that it better reflects the requests API instead of it being 
stuffed in after the fact.  And as the one responsible for that stuffing, it 
was pretty blunt and really needs to be cleaned up more than Alessio did.

only a few libs (maybe just glance and swift?) don't use requests at this point 
and I think the resistance there is the chunked transfers they both do.

There's a few more items here that are needed for glance to be able to work 
with requests (which we really really want).
1) Support for 100-expect-continue is probably going to be required in glance 
as well as swift
2) Support for turning off tls/ssl compression (our streams are already 
compressed)

I feel like we *must* have somebody here who is able and willing to add these 
features to requests, which seems like the right approach.

Let me talk to upstream about this; I know a lot of people involved. Patches 
from us probably needed, but I’ll ask



I'm really curious what 'noauth' means against APIs that have few, if any, 
calls that operate without a valid token.

dt

--

Dean Troyer
dtro...@gmail.com

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Joe Gordon]

On Thu, Jan 16, 2014 at 12:07 PM, Alexei Kornienko <
alexei.kornie...@gmail.com> wrote:

>  On 01/16/2014 06:15 PM, Jesse Noller wrote:
>
>
>  On Jan 16, 2014, at 9:54 AM, Alexei Kornienko 
> wrote:
>
>  On 01/16/2014 05:25 PM, Jesse Noller wrote:
>
>
>  On Jan 16, 2014, at 9:07 AM, Joe Gordon  wrote:
>
>
>
>
> On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller 
> wrote:
>
>>
>>   On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah 
>> wrote:
>>
>>
>> On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones  wrote:
>>
>>> Once a common library is in place, is there any intention to (or
>>> resistance against) collapsing the clients into a single project or even a
>>> single command (a la busybox)?
>>
>>
>>
>>  that's what openstackclient is here for
>> https://github.com/openstack/python-openstackclient
>>
>>
>>   After speaking with people working on OSC and looking at the code base
>> in depth; I don’t think this addresses what Chris is implying: OSC wraps
>> the individual CLIs built by each project today, instead of the inverse: a
>> common backend that the individual CLIs can wrap - the latter is an
>> important distinction as currently, building a single binary install of OSC
>> for say, Windows is difficult given the dependency tree incurred by each of
>> the wrapped CLIs, difference in dependencies, structure, etc.
>>
>>  Also, wrapping a series of inconsistent back end Client classes /
>> functions / methods means that the layer that presents a consistent user
>> interface (OSC) to the user is made more complex juggling
>> names/renames/commands/etc.
>>
>>  In the inverted case of what we have today (single backend); as a
>> developer of user interfaces (CLIs, Applications, Web apps (horizon)) you
>> would be able to:
>>
>>  from openstack.common.api import Auth
>>  from openstack.common.api import Compute
>> from openstack.common.util import cli_tools
>>
>>  my_cli = cli_tools.build(…)
>>
>>  def my_command(cli):
>> compute = Compute(Auth(cli.tentant…, connect=True))
>> compute.list_flavors()
>>
>>  This would mean that *even if the individual clients needed or wanted
>> to keep their specific CLIs, they would be able to use a not “least common
>> denominator” back end (each service can have a rich common.api.compute.pyor 
>> api.compute/client.py and extend where needed. However tools like
>> horizon / openstackclient can choose not to leverage the “power
>> user/operator/admin” components and present a simplified user interface.
>>
>>  I’m working on a wiki page + blueprint to brainstorm how we could
>> accomplish this based off of what work is in flight today (see doug’s
>> linked blueprint) and sussing out a layout / API strawman for discussion.
>>
>>  Some of the additions that came out of this email threads and others:
>>
>>  1. Common backend should provide / offer caching utilities
>> 2. Auth retries need to be handled by the auth object, and each
>> sub-project delegates to the auth object to manage that.
>> 3. Verified Mocks / Stubs / Fakes must be provided for proper unit
>> testing
>>
>
>  I am happy to see this work being done, there is definitely a lot of
> work to be done on the clients.
>
>  This blueprint sounds like its still being fleshed out, so I am
> wondering what the value is of the current patches
> https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z
>
>  Those patches mainly sync cliutils and apiutils from oslo into the
> assorted clients. But if this blueprint is about the python API and not the
> CLI (as that would be the openstack-pythonclient), why sync in apiutils?
>
>  Also does this need to go through oslo-incubator or can this start out
> as a library? Making this a library earlier on will reduce the number of
> patches needed to get 20+ repositories to use this.
>
>
>  Alexei and others have at least started the first stage of a rollout -
> the blueprint(s) needs additional work, planning and discussion, but his
> work is a good first step (reduce the duplication of code) although I am
> worried that the libraries and APIs / namespaces will need to change if we
> continue these discussions which potentially means re-doing work.
>
>  If we take a step back, a rollout process might be:
>
>  1: Solidify the libraries / layout / naming conventions (blueprint)
> 2: Solidify the APIs exposed to consumers (blueprint)
> 3: Pick up on the common-client-library-2 work which is primarily a
> migration of common code into oslo today, into the structure defined by 1 &
> 2
>
>  So, I sort of agree: moving / collapsing code now might be premature. I
> do strongly agree it should stand on its own as a library rather than an
> oslo incubator however. We should start with a single, clean namespace /
> library rather than depending on oslo directly.
>
> Knowing usual openstack workflow I'm afraid that #1,#2 with a waterfall
> approach may take years to be complete.
> And after they'll be approved it will become clear that this architecture
> is already outdated.
> We try to use iterat

Re: [openstack-dev] a "common" client library

[Mark Washenberger]

On Thu, Jan 16, 2014 at 12:03 AM, Flavio Percoco  wrote:

> On 15/01/14 21:35 +, Jesse Noller wrote:
>
>>
>> On Jan 15, 2014, at 1:37 PM, Doug Hellmann 
>> wrote:
>>
>>  Several people have mentioned to me that they are interested in, or
>>> actively working on, code related to a "common" client library -- something
>>> meant to be reused directly as a basis for creating a common library for
>>> all of the openstack clients to use. There's a blueprint [1] in oslo, and I
>>> believe the keystone devs and unified CLI teams are probably interested in
>>> ensuring that the resulting API ends up meeting all of our various
>>> requirements.
>>>
>>> If you're interested in this effort, please subscribe to the blueprint
>>> and use that to coordinate efforts so we don't produce more than one common
>>> library. ;-)
>>>
>>> Thanks,
>>> Doug
>>>
>>>
>>> [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
>>>
>>
>> *raises hand*
>>
>> Me me!
>>
>> I’ve been talking to many contributors about the Developer Experience
>> stuff I emailed out prior to the holidays and I was starting blueprint
>> work, but this is a great pointer. I’m going to have to sync up with Alexei.
>>
>> I think solving this for openstack developers and maintainers as the
>> blueprint says is a big win in terms of code reuse / maintenance and
>> consistent but more so for *end-user developers* consuming openstack clouds.
>>
>> Some background - there’s some terminology mismatch but the rough idea is
>> the same:
>>
>> * A centralized “SDK” (Software Development Kit) would be built
>> condensing the common code and logic and operations into a single namespace.
>>
>> * This SDK would be able to be used by “downstream” CLIs - essentially
>> the CLIs become a specialized front end - and in some cases, only an
>> argparse or cliff front-end to the SDK methods located in the (for example)
>> openstack.client.api.compute
>>
>> * The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived
>> clients - all of the openstack.client.api.** classes would accept an Auth
>> object to delegate management / mocking of the Auth / service catalog stuff
>> to. This means developers building applications (say for example, horizon)
>> don’t need to worry about token/expired authentication/etc.
>>
>> * Simplify the dependency graph & code for the existing tools to enable
>> single binary installs (py2exe, py2app, etc) for end users of the command
>> line tools.
>>
>> Short version: if a developer wants to consume an openstack cloud; the
>> would have a single SDK with minimal dependencies and import from a single
>> namespace. An example application might look like:
>>
>> from openstack.api import AuthV2
>> from openstack.api import ComputeV2
>>
>> myauth = AuthV2(…., connect=True)
>> compute = ComputeV2(myauth)
>>
>> compute.list_flavors()
>>
>>
> I know this is an example but, could we leave the version out of the
> class name? Having something like:
>
> from openstack.api.v2 import Compute
>
>or
>
> from openstack.compute.v2 import Instance
>
> (just made that up)
>
> for marconi we're using the later.


Just throwing this out there because it seems relevant to client design.

As we've been looking at porting clients to using v2 of the Images API, its
seems more and more to me that including the *server* version in the main
import path is a real obstacle.

IMO any future client libs should write library interfaces based on the
peculiarities of user needs, not based on the vagaries of the server
version. So as a user of this library I would do something like:

  1 from openstack.api import images
  2 client = images.make_me_a_client(auth_url, etcetera) # all version
negotiation is happening here
  3 client.list_images()  # works more or less same no matter who I'm
talking to

Now, there would still likely be hidden implementation code that is
different per server version and which is instantiated in line 2 above, and
maybe that's the library path stuff you are talking about.


>
>
>  This greatly improves the developer experience both internal to openstack
>> and externally. Currently OpenStack has 22+ (counting stackforge) potential
>> libraries a developer may need to install to use a full deployment of
>> OpenStack:
>>
>>  * python-keystoneclient (identity)
>>  * python-glanceclient (image)
>>  * python-novaclient (compute)
>>  * python-troveclient (database)
>>  * python-neutronclient (network)
>>  * python-ironicclient (bare metal)
>>  * python-heatclient (orchestration)
>>  * python-cinderclient (block storage)
>>  * python-ceilometerclient (telemetry, metrics & billing)
>>  * python-swiftclient (object storage)
>>  * python-savannaclient (big data)
>>  * python-openstackclient (meta client package)
>>  * python-marconiclient (queueing)
>>  * python-tuskarclient (tripleo / management)
>>  * python-melangeclient (dead)
>>  * python-barbicanclient (secrets)
>>  * python-solumclient (ALM)
>>  * python-muranoclient (applica

Re: [openstack-dev] a "common" client library

[Mark Washenberger]

On Wed, Jan 15, 2014 at 7:53 PM, Alexei Kornienko <
alexei.kornie...@gmail.com> wrote:

> >I did notice, however, that neutronclient is
> conspicuously absent from the Work Items in the blueprint's Whiteboard.
> It will surely be added later. We already working on several things in
> parallel and we will add neutronclient soon.
>
>
> >I would love to see a bit more detail on the structure of the lib(s), the
> blueprint really doesn't discuss the design/organization/intended API of
> the libs.  For example, I would hope the distinction between the various
> layers of a client stack don't get lost, i.e. not mixing the low-level REST
> API bits with the higher-level CLI parsers and decorators.
> >Does the long-term goals include a common caching layer?
>
> Distinction between client layers won't get lost and would only be
> improved. My basic idea is the following:
> 1) Transport layer would handle all transport related stuff - HTTP, JSON
> encoding, auth, caching, etc.
> 2) Model layer (Resource classes, BaseManager, etc.) will handle data
> representation, validation
> 3) API layer will handle all project specific stuff - url mapping, etc.
> (This will be imported to use client in other applications)
> 4) Cli level will handle all stuff related to cli mapping - argparse,
> argcomplete, etc.
>

I'm really excited about this. I think consolidating layers 1 and 4 will be
a huge benefit for deployers and users.

I'm hoping we can structure layers 2 and 3 a bit flexibly to allow for
existing project differences and proper ownership. For example, in Glance
we use jsonschema somewhat so our validation is a bit different. Also, I
consider the definition of resources and url mappings for images to be
something that should be owned by the Images program. I'm confident,
however, that we can figure out how to structure the libraries,
deliverables, and process to reflect that ownership.


>
> >I believe the current effort referenced by the blueprint is focusing on
> moving existing code into the incubator for reuse, to make it easier to
> restructure later. Alexei, do I have that correct?
> You are right. The first thing we do is try to make all clients look/work
> in similar way. After we'll continue our work with improving overall
> structure.
>
>
>
>
> 2014/1/16 Noorul Islam K M 
>
>> Doug Hellmann  writes:
>>
>> > Several people have mentioned to me that they are interested in, or
>> > actively working on, code related to a "common" client library --
>> something
>> > meant to be reused directly as a basis for creating a common library for
>> > all of the openstack clients to use. There's a blueprint [1] in oslo,
>> and I
>> > believe the keystone devs and unified CLI teams are probably interested
>> in
>> > ensuring that the resulting API ends up meeting all of our various
>> > requirements.
>> >
>> > If you're interested in this effort, please subscribe to the blueprint
>> and
>> > use that to coordinate efforts so we don't produce more than one common
>> > library. ;-)
>> >
>>
>> Solum is already using it https://review.openstack.org/#/c/58067/
>>
>> I would love to watch this space.
>>
>> Regards,
>> Noorul
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Mark Washenberger]

On Thu, Jan 16, 2014 at 8:06 AM, Dean Troyer  wrote:

> On Thu, Jan 16, 2014 at 9:37 AM, Jesse Noller 
> wrote:
>
>> On Jan 16, 2014, at 9:26 AM, Justin Hammond 
>> wrote:
>>
>> I'm not sure if it was said, but which httplib using being used (urllib3
>> maybe?). Also I noticed many people were talking about supporting auth
>> properly, but are there any intentions to properly support 'noauth'
>> (python-neutronclient, for instance, doesn't support it properly as of
>> this writing)?
>>
>> Can you detail out noauth for me; and I would say the defacto httplib in
>> python today is python-requests - urllib3 is also good but I would say from
>> a *consumer* standpoint requests offers more in terms of usability /
>> extensibility
>>
>
> requests is built on top of urllib3 so there's that...
>
> The biggest reaon I favor using Jamie Lennox's new session layer stuff in
> keystoneclient is that it better reflects the requests API instead of it
> being stuffed in after the fact.  And as the one responsible for that
> stuffing, it was pretty blunt and really needs to be cleaned up more than
> Alessio did.
>
> only a few libs (maybe just glance and swift?) don't use requests at this
> point and I think the resistance there is the chunked transfers they both
> do.
>

There's a few more items here that are needed for glance to be able to work
with requests (which we really really want).
1) Support for 100-expect-continue is probably going to be required in
glance as well as swift
2) Support for turning off tls/ssl compression (our streams are already
compressed)

I feel like we *must* have somebody here who is able and willing to add
these features to requests, which seems like the right approach.


>
> I'm really curious what 'noauth' means against APIs that have few, if any,
> calls that operate without a valid token.
>
> dt
>
> --
>
> Dean Troyer
> dtro...@gmail.com
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

On 01/16/2014 06:15 PM, Jesse Noller wrote:


On Jan 16, 2014, at 9:54 AM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:



On 01/16/2014 05:25 PM, Jesse Noller wrote:


On Jan 16, 2014, at 9:07 AM, Joe Gordon > wrote:






On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:



On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah
mailto:chmo...@enovance.com>> wrote:



On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones mailto:c...@tenshu.net>> wrote:

Once a common library is in place, is there any intention
to (or resistance against) collapsing the clients into a
single project or even a single command (a la busybox)?



that's what openstackclient is here for
https://github.com/openstack/python-openstackclient


After speaking with people working on OSC and looking at the
code base in depth; I don't think this addresses what Chris is
implying: OSC wraps the individual CLIs built by each project
today, instead of the inverse: a common backend that the
individual CLIs can wrap - the latter is an important
distinction as currently, building a single binary install of
OSC for say, Windows is difficult given the dependency tree
incurred by each of the wrapped CLIs, difference in
dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client classes
/ functions / methods means that the layer that presents a
consistent user interface (OSC) to the user is made more
complex juggling names/renames/commands/etc.

In the inverted case of what we have today (single backend); as
a developer of user interfaces (CLIs, Applications, Web apps
(horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(...)

def my_command(cli):
compute = Compute(Auth(cli.tentant..., connect=True))
compute.list_flavors()

This would mean that *even if the individual clients needed or
wanted to keep their specific CLIs, they would be able to use a
not "least common denominator" back end (each service can have
a rich common.api.compute.py  or
api.compute/client.py and extend where needed. However tools
like horizon / openstackclient can choose not to leverage the
"power user/operator/admin" components and present a simplified
user interface.

I'm working on a wiki page + blueprint to brainstorm how we
could accomplish this based off of what work is in flight today
(see doug's linked blueprint) and sussing out a layout / API
strawman for discussion.

Some of the additions that came out of this email threads and
others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and each
sub-project delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for proper
unit testing


I am happy to see this work being done, there is definitely a lot 
of work to be done on the clients.


This blueprint sounds like its still being fleshed out, so I am 
wondering what the value is of the current patches 
https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z


Those patches mainly sync cliutils and apiutils from oslo into the 
assorted clients. But if this blueprint is about the python API and 
not the CLI (as that would be the openstack-pythonclient), why sync 
in apiutils?


Also does this need to go through oslo-incubator or can this start 
out as a library? Making this a library earlier on will reduce the 
number of patches needed to get 20+ repositories to use this.




Alexei and others have at least started the first stage of a rollout 
- the blueprint(s) needs additional work, planning and discussion, 
but his work is a good first step (reduce the duplication of code) 
although I am worried that the libraries and APIs / namespaces will 
need to change if we continue these discussions which potentially 
means re-doing work.


If we take a step back, a rollout process might be:

1: Solidify the libraries / layout / naming conventions (blueprint)
2: Solidify the APIs exposed to consumers (blueprint)
3: Pick up on the common-client-library-2 work which is primarily a 
migration of common code into oslo today, into the structure defined 
by 1 & 2


So, I sort of agree: moving / collapsing code now might be 
premature. I do strongly agree it should stand on its own as a 
library rather than an oslo incubator however. We should start with 
a single, clean namespace / library rather than depending on oslo 
directly.
Knowing usual openstack workflow I'm afraid that #1,#2 with a 
waterfall approach may take years to be complete.
And after they'll be approved it will become clear that this 
a

Re: [openstack-dev] a "common" client library

[Joe Gordon]

On Thu, Jan 16, 2014 at 10:25 AM, Jesse Noller
wrote:

>
>  On Jan 16, 2014, at 9:07 AM, Joe Gordon  wrote:
>
>
>
>
> On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller 
> wrote:
>
>>
>>   On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah 
>> wrote:
>>
>>
>> On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones  wrote:
>>
>>> Once a common library is in place, is there any intention to (or
>>> resistance against) collapsing the clients into a single project or even a
>>> single command (a la busybox)?
>>
>>
>>
>>  that's what openstackclient is here for
>> https://github.com/openstack/python-openstackclient
>>
>>
>>   After speaking with people working on OSC and looking at the code base
>> in depth; I don’t think this addresses what Chris is implying: OSC wraps
>> the individual CLIs built by each project today, instead of the inverse: a
>> common backend that the individual CLIs can wrap - the latter is an
>> important distinction as currently, building a single binary install of OSC
>> for say, Windows is difficult given the dependency tree incurred by each of
>> the wrapped CLIs, difference in dependencies, structure, etc.
>>
>>  Also, wrapping a series of inconsistent back end Client classes /
>> functions / methods means that the layer that presents a consistent user
>> interface (OSC) to the user is made more complex juggling
>> names/renames/commands/etc.
>>
>>  In the inverted case of what we have today (single backend); as a
>> developer of user interfaces (CLIs, Applications, Web apps (horizon)) you
>> would be able to:
>>
>>  from openstack.common.api import Auth
>>  from openstack.common.api import Compute
>> from openstack.common.util import cli_tools
>>
>>  my_cli = cli_tools.build(…)
>>
>>  def my_command(cli):
>> compute = Compute(Auth(cli.tentant…, connect=True))
>> compute.list_flavors()
>>
>>  This would mean that *even if the individual clients needed or wanted
>> to keep their specific CLIs, they would be able to use a not “least common
>> denominator” back end (each service can have a rich common.api.compute.pyor 
>> api.compute/client.py and extend where needed. However tools like
>> horizon / openstackclient can choose not to leverage the “power
>> user/operator/admin” components and present a simplified user interface.
>>
>>  I’m working on a wiki page + blueprint to brainstorm how we could
>> accomplish this based off of what work is in flight today (see doug’s
>> linked blueprint) and sussing out a layout / API strawman for discussion.
>>
>>  Some of the additions that came out of this email threads and others:
>>
>>  1. Common backend should provide / offer caching utilities
>> 2. Auth retries need to be handled by the auth object, and each
>> sub-project delegates to the auth object to manage that.
>> 3. Verified Mocks / Stubs / Fakes must be provided for proper unit
>> testing
>>
>
>  I am happy to see this work being done, there is definitely a lot of
> work to be done on the clients.
>
>  This blueprint sounds like its still being fleshed out, so I am
> wondering what the value is of the current patches
> https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z
>
>  Those patches mainly sync cliutils and apiutils from oslo into the
> assorted clients. But if this blueprint is about the python API and not the
> CLI (as that would be the openstack-pythonclient), why sync in apiutils?
>
>  Also does this need to go through oslo-incubator or can this start out
> as a library? Making this a library earlier on will reduce the number of
> patches needed to get 20+ repositories to use this.
>
>
>  Alexei and others have at least started the first stage of a rollout -
> the blueprint(s) needs additional work, planning and discussion, but his
> work is a good first step (reduce the duplication of code) although I am
> worried that the libraries and APIs / namespaces will need to change if we
> continue these discussions which potentially means re-doing work.
>
>  If we take a step back, a rollout process might be:
>
>  1: Solidify the libraries / layout / naming conventions (blueprint)
> 2: Solidify the APIs exposed to consumers (blueprint)
> 3: Pick up on the common-client-library-2 work which is primarily a
> migration of common code into oslo today, into the structure defined by 1 &
> 2
>
>  So, I sort of agree: moving / collapsing code now might be premature. I
> do strongly agree it should stand on its own as a library rather than an
> oslo incubator however. We should start with a single, clean namespace /
> library rather than depending on oslo directly.
>

Sounds good to me, thanks.


>
>  jesse
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Dean Troyer]

On Thu, Jan 16, 2014 at 10:23 AM, Jay Pipes  wrote:
>
> Right, but requests supports chunked-transfer encoding properly, so
> really there's no reason those clients could not move to a
> requests-based codebase.
>

Absolutely...it was totally me chickening out at the time why they didn't
get changed.  I feel a bit braver now...  :)

>
dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Chmouel Boudjnah]

On Thu, Jan 16, 2014 at 5:23 PM, Jay Pipes  wrote:

> Right, but requests supports chunked-transfer encoding properly, so
> really there's no reason those clients could not move to a
> requests-based codebase.
>

We had that discussion for swiftclient and we are not against it but
unfortunately there is no way to disable compression using requests which
is a requirement for performances with swift (and security).

see: https://review.openstack.org/#/c/33473/

and: https://github.com/kennethreitz/requests/issues/1853

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jay Pipes]

On Thu, 2014-01-16 at 10:06 -0600, Dean Troyer wrote:
> On Thu, Jan 16, 2014 at 9:37 AM, Jesse Noller
>  wrote:
> On Jan 16, 2014, at 9:26 AM, Justin Hammond
>  wrote:
> > I'm not sure if it was said, but which httplib using being
> > used (urllib3
> > maybe?). Also I noticed many people were talking about
> > supporting auth
> > properly, but are there any intentions to properly support
> > 'noauth'
> > (python-neutronclient, for instance, doesn't support it
> > properly as of
> > this writing)? 
> > 
> Can you detail out noauth for me; and I would say the defacto
> httplib in python today is python-requests - urllib3 is also
> good but I would say from a *consumer* standpoint requests
> offers more in terms of usability / extensibility 
> 
> 
> requests is built on top of urllib3 so there's that...
> 
> 
> The biggest reaon I favor using Jamie Lennox's new session layer stuff
> in keystoneclient is that it better reflects the requests API instead
> of it being stuffed in after the fact.  And as the one responsible for
> that stuffing, it was pretty blunt and really needs to be cleaned up
> more than Alessio did.
> 
> 
> only a few libs (maybe just glance and swift?) don't use requests at
> this point and I think the resistance there is the chunked transfers
> they both do.

Right, but requests supports chunked-transfer encoding properly, so
really there's no reason those clients could not move to a
requests-based codebase.

Best,
-jay

> 
> I'm really curious what 'noauth' means against APIs that have few, if
> any, calls that operate without a valid token.
> 
> 
> dt
> 
> 
> -- 
> 
> Dean Troyer
> dtro...@gmail.com
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Dean Troyer]

On Thu, Jan 16, 2014 at 9:54 AM, Alexei Kornienko <
alexei.kornie...@gmail.com> wrote:
>
> Knowing usual openstack workflow I'm afraid that #1,#2 with a waterfall
> approach may take years to be complete.
> And after they'll be approved it will become clear that this architecture
> is already outdated.
> We try to use iterative approach for clients refactoring.
> We started our work from removing code duplication because it already
> gives a direct positive effect on client projects.
> If you can show us better way of moving forward please help us by
> uploading patches on this topic.
>
> Talk is cheap. Show me the code. (c) Linus
>

python-keystoneclient has the Session bits commiited, and the auth bits in
flight.  So we'll all be using it one way or another.

OSC has a very-slimmed-down model for replacing the REST and API layers
already present underneath the object-store commands as an experiment to
see how slim I could make then and still be usable.   It isn't necessarily
what I want to ship everywhere, but the REST layer is eerily similar to
Jamie's in keystoneclient.  There's why my bias...

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Justin Hammond]

My prioritization of noauth is rooted in the fact that we're finding that
the current pattern of hitting auth to validate a token is not scaling
well. Out current solution to this scale issue is:

- use noauth when possible between the services
- use normal auth for public services
- provide a method to create a 'trusted environment'

While this problem may not be prevalent in other deployments I will add
that support noauth in the client 'just makes sense' when the services
themselves support them.

For instance our setup looks like:

User -> Auth to Nova -> Nova/Computes -> NoAuth to neutron in 'trusted
environment'

It saves quite a few calls to identity in this way and scales a lot better.

On 1/16/14 11:06 AM, "Dean Troyer"  wrote:

>On Thu, Jan 16, 2014 at 9:37 AM, Jesse Noller
> wrote:
>
>On Jan 16, 2014, at 9:26 AM, Justin Hammond
> wrote:
>
>
>I'm not sure if it was said, but which httplib using being used (urllib3
>maybe?). Also I noticed many people were talking about supporting auth
>properly, but are there any intentions to properly support 'noauth'
>(python-neutronclient, for instance, doesn't support it properly as of
>this writing)? 
>
>
>
>
>Can you detail out noauth for me; and I would say the defacto httplib in
>python today is python-requests - urllib3 is also good but I would say
>from a *consumer* standpoint requests offers more in terms of usability /
>extensibility 
>
>
>
>
>
>
>requests is built on top of urllib3 so there's that...
>
>The biggest reaon I favor using Jamie Lennox's new session layer stuff in
>keystoneclient is that it better reflects the requests API instead of it
>being stuffed in after the fact.  And as the one responsible for that
>stuffing, it was pretty blunt and really needs to be cleaned up more than
>Alessio did.
>
>only a few libs (maybe just glance and swift?) don't use requests at this
>point and I think the resistance there is the chunked transfers they both
>do.
>
>I'm really curious what 'noauth' means against APIs that have few, if
>any, calls that operate without a valid token.
>
>dt
>
>-- 
>
>Dean Troyer
>dtro...@gmail.com
>
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 9:54 AM, Alexei Kornienko 
mailto:alexei.kornie...@gmail.com>> wrote:

On 01/16/2014 05:25 PM, Jesse Noller wrote:

On Jan 16, 2014, at 9:07 AM, Joe Gordon 
mailto:joe.gord...@gmail.com>> wrote:




On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:

On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah 
mailto:chmo...@enovance.com>> wrote:


On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones 
mailto:c...@tenshu.net>> wrote:
Once a common library is in place, is there any intention to (or resistance 
against) collapsing the clients into a single project or even a single command 
(a la busybox)?


that's what openstackclient is here for 
https://github.com/openstack/python-openstackclient

After speaking with people working on OSC and looking at the code base in 
depth; I don’t think this addresses what Chris is implying: OSC wraps the 
individual CLIs built by each project today, instead of the inverse: a common 
backend that the individual CLIs can wrap - the latter is an important 
distinction as currently, building a single binary install of OSC for say, 
Windows is difficult given the dependency tree incurred by each of the wrapped 
CLIs, difference in dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client classes / functions / 
methods means that the layer that presents a consistent user interface (OSC) to 
the user is made more complex juggling names/renames/commands/etc.

In the inverted case of what we have today (single backend); as a developer of 
user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(…)

def my_command(cli):
compute = Compute(Auth(cli.tentant…, connect=True))
compute.list_flavors()

This would mean that *even if the individual clients needed or wanted to keep 
their specific CLIs, they would be able to use a not “least common denominator” 
back end (each service can have a rich 
common.api.compute.py or api.compute/client.py 
and extend where needed. However tools like horizon / openstackclient can 
choose not to leverage the “power user/operator/admin” components and present a 
simplified user interface.

I’m working on a wiki page + blueprint to brainstorm how we could accomplish 
this based off of what work is in flight today (see doug’s linked blueprint) 
and sussing out a layout / API strawman for discussion.

Some of the additions that came out of this email threads and others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and each sub-project 
delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing

I am happy to see this work being done, there is definitely a lot of work to be 
done on the clients.

This blueprint sounds like its still being fleshed out, so I am wondering what 
the value is of the current patches 
https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z

Those patches mainly sync cliutils and apiutils from oslo into the assorted 
clients. But if this blueprint is about the python API and not the CLI (as that 
would be the openstack-pythonclient), why sync in apiutils?

Also does this need to go through oslo-incubator or can this start out as a 
library? Making this a library earlier on will reduce the number of patches 
needed to get 20+ repositories to use this.


Alexei and others have at least started the first stage of a rollout - the 
blueprint(s) needs additional work, planning and discussion, but his work is a 
good first step (reduce the duplication of code) although I am worried that the 
libraries and APIs / namespaces will need to change if we continue these 
discussions which potentially means re-doing work.

If we take a step back, a rollout process might be:

1: Solidify the libraries / layout / naming conventions (blueprint)
2: Solidify the APIs exposed to consumers (blueprint)
3: Pick up on the common-client-library-2 work which is primarily a migration 
of common code into oslo today, into the structure defined by 1 & 2

So, I sort of agree: moving / collapsing code now might be premature. I do 
strongly agree it should stand on its own as a library rather than an oslo 
incubator however. We should start with a single, clean namespace / library 
rather than depending on oslo directly.
Knowing usual openstack workflow I'm afraid that #1,#2 with a waterfall 
approach may take years to be complete.
And after they'll be approved it will become clear that this architecture is 
already outdated.
We try to use iterative approach for clients refactoring.
We started our work from removing code duplication because it already gives a 
direct positive effect on client projects.
If you can show us better way of moving forward pl

Re: [openstack-dev] a "common" client library

[Dean Troyer]

On Thu, Jan 16, 2014 at 8:45 AM, Jesse Noller wrote:

> After speaking with people working on OSC and looking at the code base in
> depth; I don’t think this addresses what Chris is implying: OSC wraps the
> individual CLIs built by each project today, instead of the inverse: a
> common backend that the individual CLIs can wrap - the latter is an
> important distinction as currently, building a single binary install of OSC
> for say, Windows is difficult given the dependency tree incurred by each of
> the wrapped CLIs, difference in dependencies, structure, etc.
>

OSC is the top of the cake and was the most beneficial to user experience
so it went first.  I would love to consume fewer libraries and
dependencies, so much that I still have a project to do this in a language
that can easily ship a single binary client.

What I think we're talking about here is the bottom of the cake and
eliminating duplication and accumulated cruft from repeated forking and
later direction changes.

The creamy gooey middle is the API-specific bits that right stay exactly
where they are (for now??) and can continue to ship a stand-alone cli if
they wish.

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Dean Troyer]

On Thu, Jan 16, 2014 at 9:37 AM, Jesse Noller wrote:

> On Jan 16, 2014, at 9:26 AM, Justin Hammond 
> wrote:
>
> I'm not sure if it was said, but which httplib using being used (urllib3
> maybe?). Also I noticed many people were talking about supporting auth
> properly, but are there any intentions to properly support 'noauth'
> (python-neutronclient, for instance, doesn't support it properly as of
> this writing)?
>
> Can you detail out noauth for me; and I would say the defacto httplib in
> python today is python-requests - urllib3 is also good but I would say from
> a *consumer* standpoint requests offers more in terms of usability /
> extensibility
>

requests is built on top of urllib3 so there's that...

The biggest reaon I favor using Jamie Lennox's new session layer stuff in
keystoneclient is that it better reflects the requests API instead of it
being stuffed in after the fact.  And as the one responsible for that
stuffing, it was pretty blunt and really needs to be cleaned up more than
Alessio did.

only a few libs (maybe just glance and swift?) don't use requests at this
point and I think the resistance there is the chunked transfers they both
do.

I'm really curious what 'noauth' means against APIs that have few, if any,
calls that operate without a valid token.

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Doug Hellmann]

On Wed, Jan 15, 2014 at 4:35 PM, Jesse Noller wrote:

>
> On Jan 15, 2014, at 1:37 PM, Doug Hellmann 
> wrote:
>
> > Several people have mentioned to me that they are interested in, or
> actively working on, code related to a "common" client library -- something
> meant to be reused directly as a basis for creating a common library for
> all of the openstack clients to use. There's a blueprint [1] in oslo, and I
> believe the keystone devs and unified CLI teams are probably interested in
> ensuring that the resulting API ends up meeting all of our various
> requirements.
> >
> > If you're interested in this effort, please subscribe to the blueprint
> and use that to coordinate efforts so we don't produce more than one common
> library. ;-)
> >
> > Thanks,
> > Doug
> >
> >
> > [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
>
> *raises hand*
>
> Me me!
>
> I’ve been talking to many contributors about the Developer Experience
> stuff I emailed out prior to the holidays and I was starting blueprint
> work, but this is a great pointer. I’m going to have to sync up with Alexei.
>
> I think solving this for openstack developers and maintainers as the
> blueprint says is a big win in terms of code reuse / maintenance and
> consistent but more so for *end-user developers* consuming openstack clouds.
>
> Some background - there’s some terminology mismatch but the rough idea is
> the same:
>
> * A centralized “SDK” (Software Development Kit) would be built condensing
> the common code and logic and operations into a single namespace.
>
> * This SDK would be able to be used by “downstream” CLIs - essentially the
> CLIs become a specialized front end - and in some cases, only an argparse
> or cliff front-end to the SDK methods located in the (for example)
> openstack.client.api.compute
>
> * The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived
> clients - all of the openstack.client.api.** classes would accept an Auth
> object to delegate management / mocking of the Auth / service catalog stuff
> to. This means developers building applications (say for example, horizon)
> don’t need to worry about token/expired authentication/etc.
>
> * Simplify the dependency graph & code for the existing tools to enable
> single binary installs (py2exe, py2app, etc) for end users of the command
> line tools.
>
> Short version: if a developer wants to consume an openstack cloud; the
> would have a single SDK with minimal dependencies and import from a single
> namespace. An example application might look like:
>
> from openstack.api import AuthV2
> from openstack.api import ComputeV2
>
> myauth = AuthV2(…., connect=True)
> compute = ComputeV2(myauth)
>
> compute.list_flavors()
>
> This greatly improves the developer experience both internal to openstack
> and externally. Currently OpenStack has 22+ (counting stackforge) potential
> libraries a developer may need to install to use a full deployment of
> OpenStack:
>
>   * python-keystoneclient (identity)
>   * python-glanceclient (image)
>   * python-novaclient (compute)
>   * python-troveclient (database)
>   * python-neutronclient (network)
>   * python-ironicclient (bare metal)
>   * python-heatclient (orchestration)
>   * python-cinderclient (block storage)
>   * python-ceilometerclient (telemetry, metrics & billing)
>   * python-swiftclient (object storage)
>   * python-savannaclient (big data)
>   * python-openstackclient (meta client package)
>   * python-marconiclient (queueing)
>   * python-tuskarclient (tripleo / management)
>   * python-melangeclient (dead)
>   * python-barbicanclient (secrets)
>   * python-solumclient (ALM)
>   * python-muranoclient (application catalog)
>   * python-manilaclient (shared filesystems)
>   * python-libraclient (load balancers)
>   * python-climateclient (reservations)
>   * python-designateclient (Moniker/DNS)
>
> If you exclude the above and look on PyPI:
>
> On PyPi (client libraries/SDKs only, excluding the above - not maintained
> by openstack):
>
>  * hpcloud-auth-openstack 1.0
>  * python-openstackclient 0.2.2
>  * rackspace-auth-openstack 1.1
>  * posthaste 0.2.2
>  * pyrax 1.6.2
>  * serverherald 0.0.1
>  * warm 0.3.1
>  * vaporize 0.3.2
>  * swiftsc (https://github.com/mkouhei/swiftsc)
>  * bookofnova 0.007
>  * nova-adminclient 0.1.8
>  * python-quantumclient 2.2.4.3
>  * python-stackhelper 0.0.7.1.gcab1eb0
>  * swift-bench 1.0
>  * swiftly 1.12
>  * txAWS 0.2.3
>  * cfupload 0.5.1
>  * python-reddwarfclient 0.1.2
>  * python-automationclient 1.2.1
>  * rackspace-glanceclient 0.9
>  * rackspace-novaclient 1.4
>
> If you ignore PyPI and just want to install the base say - 7 services,
> each one of the python-** clients has its own dependency tree and may or
> may not build from one of the others. If a vendor wants to extend any of
> them, it’s basically a fork instead of a clean plugin system.
>
> On the CLI front - this would *greatly* simplify the work openstackclient
> needs to do - it w

Re: [openstack-dev] a "common" client library

[Chmouel Boudjnah]

On Thu, Jan 16, 2014 at 4:37 PM, Jesse Noller wrote:

> Can you detail out noauth for me; and I would say the defacto httplib in
> python today is python-requests - urllib3 is also good but I would say from
> a *consumer* standpoint requests offers more in terms of usability /
> extensibility
>

FYI: python-requests is using urllib3 for its backend (which use httplib
after that).

Chmouel.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Justin Hammond]

I'm not sure if it was said, but which httplib using being used (urllib3
maybe?). Also I noticed many people were talking about supporting auth
properly, but are there any intentions to properly support 'noauth'
(python-neutronclient, for instance, doesn't support it properly as of
this writing)? 

On 1/15/14 10:53 PM, "Alexei Kornienko"  wrote:

>>I did notice, however, that neutronclient is
>conspicuously absent from the Work Items in the blueprint's Whiteboard.
>
>It will surely be added later. We already working on several things in
>parallel and we will add neutronclient soon.

Do you need another person to make neutron client?

>
>>I would love to see a bit more detail on the structure of the
>lib(s), the blueprint really doesn't discuss the
>design/organization/intended API of the libs.  For example, I would hope
> the distinction between the various layers of a client stack don't get
>lost, i.e. not mixing the low-level REST API bits with the higher-level
>CLI parsers and decorators.
>>Does the long-term goals include a common caching layer?
>
>
>Distinction between client layers won't get lost and would only be
>improved. My basic idea is the following:
>
>1) Transport layer would handle all transport related stuff - HTTP, JSON
>encoding, auth, caching, etc.
>
>2) Model layer (Resource classes, BaseManager, etc.) will handle data
>representation, validation
>
>3) API layer will handle all project specific stuff - url mapping, etc.
>(This will be imported to use client in other applications)
>
>4) Cli level will handle all stuff related to cli mapping - argparse,
>argcomplete, etc.
>
>
>>I believe the current effort referenced by the blueprint is focusing on
>moving existing code into the incubator for reuse, to make it easier to
>restructure later. Alexei, do I have that correct?
>
>You are right. The first thing we do is try to make all clients look/work
>in similar way. After we'll continue our work with improving overall
>structure.
>
>
>
>
>
>
>2014/1/16 Noorul Islam K M 
>
>Doug Hellmann  writes:
>
>> Several people have mentioned to me that they are interested in, or
>> actively working on, code related to a "common" client library --
>>something
>> meant to be reused directly as a basis for creating a common library for
>> all of the openstack clients to use. There's a blueprint [1] in oslo,
>>and I
>> believe the keystone devs and unified CLI teams are probably interested
>>in
>> ensuring that the resulting API ends up meeting all of our various
>> requirements.
>>
>> If you're interested in this effort, please subscribe to the blueprint
>>and
>> use that to coordinate efforts so we don't produce more than one common
>> library. ;-)
>>
>
>
>Solum is already using it https://review.openstack.org/#/c/58067/
>
>I would love to watch this space.
>
>Regards,
>Noorul
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
>
>
>
>___
>OpenStack-dev mailing list
>OpenStack-dev@lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jay Pipes]

On Thu, 2014-01-16 at 09:03 +0100, Flavio Percoco wrote:
> On 15/01/14 21:35 +, Jesse Noller wrote:
> >
> >On Jan 15, 2014, at 1:37 PM, Doug Hellmann  
> >wrote:
> >
> >> Several people have mentioned to me that they are interested in, or 
> >> actively working on, code related to a "common" client library -- 
> >> something meant to be reused directly as a basis for creating a common 
> >> library for all of the openstack clients to use. There's a blueprint [1] 
> >> in oslo, and I believe the keystone devs and unified CLI teams are 
> >> probably interested in ensuring that the resulting API ends up meeting all 
> >> of our various requirements.
> >>
> >> If you're interested in this effort, please subscribe to the blueprint and 
> >> use that to coordinate efforts so we don't produce more than one common 
> >> library. ;-)
> >>
> >> Thanks,
> >> Doug
> >>
> >>
> >> [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
> >
> >*raises hand*
> >
> >Me me!
> >
> >I’ve been talking to many contributors about the Developer Experience stuff 
> >I emailed out prior to the holidays and I was starting blueprint work, but 
> >this is a great pointer. I’m going to have to sync up with Alexei.
> >
> >I think solving this for openstack developers and maintainers as the 
> >blueprint says is a big win in terms of code reuse / maintenance and 
> >consistent but more so for *end-user developers* consuming openstack clouds.
> >
> >Some background - there’s some terminology mismatch but the rough idea is 
> >the same:
> >
> >* A centralized “SDK” (Software Development Kit) would be built condensing 
> >the common code and logic and operations into a single namespace.
> >
> >* This SDK would be able to be used by “downstream” CLIs - essentially the 
> >CLIs become a specialized front end - and in some cases, only an argparse or 
> >cliff front-end to the SDK methods located in the (for example) 
> >openstack.client.api.compute
> >
> >* The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived 
> >clients - all of the openstack.client.api.** classes would accept an Auth 
> >object to delegate management / mocking of the Auth / service catalog stuff 
> >to. This means developers building applications (say for example, horizon) 
> >don’t need to worry about token/expired authentication/etc.
> >
> >* Simplify the dependency graph & code for the existing tools to enable 
> >single binary installs (py2exe, py2app, etc) for end users of the command 
> >line tools.
> >
> >Short version: if a developer wants to consume an openstack cloud; the would 
> >have a single SDK with minimal dependencies and import from a single 
> >namespace. An example application might look like:
> >
> >from openstack.api import AuthV2
> >from openstack.api import ComputeV2
> >
> >myauth = AuthV2(…., connect=True)
> >compute = ComputeV2(myauth)
> >
> >compute.list_flavors()
> >
> 
> I know this is an example but, could we leave the version out of the
> class name? Having something like:
> 
> from openstack.api.v2 import Compute
> 
> or
> 
> from openstack.compute.v2 import Instance
> 
> (just made that up)
> 
> for marconi we're using the later.

++

-jay


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

On 01/16/2014 05:25 PM, Jesse Noller wrote:


On Jan 16, 2014, at 9:07 AM, Joe Gordon > wrote:






On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:



On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah
mailto:chmo...@enovance.com>> wrote:



On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones mailto:c...@tenshu.net>> wrote:

Once a common library is in place, is there any intention to
(or resistance against) collapsing the clients into a single
project or even a single command (a la busybox)?



that's what openstackclient is here for
https://github.com/openstack/python-openstackclient


After speaking with people working on OSC and looking at the code
base in depth; I don't think this addresses what Chris is
implying: OSC wraps the individual CLIs built by each project
today, instead of the inverse: a common backend that the
individual CLIs can wrap - the latter is an important distinction
as currently, building a single binary install of OSC for say,
Windows is difficult given the dependency tree incurred by each
of the wrapped CLIs, difference in dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client classes /
functions / methods means that the layer that presents a
consistent user interface (OSC) to the user is made more complex
juggling names/renames/commands/etc.

In the inverted case of what we have today (single backend); as a
developer of user interfaces (CLIs, Applications, Web apps
(horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(...)

def my_command(cli):
compute = Compute(Auth(cli.tentant..., connect=True))
compute.list_flavors()

This would mean that *even if the individual clients needed or
wanted to keep their specific CLIs, they would be able to use a
not "least common denominator" back end (each service can have a
rich common.api.compute.py  or
api.compute/client.py and extend where needed. However tools like
horizon / openstackclient can choose not to leverage the "power
user/operator/admin" components and present a simplified user
interface.

I'm working on a wiki page + blueprint to brainstorm how we could
accomplish this based off of what work is in flight today (see
doug's linked blueprint) and sussing out a layout / API strawman
for discussion.

Some of the additions that came out of this email threads and others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and each
sub-project delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for proper
unit testing


I am happy to see this work being done, there is definitely a lot of 
work to be done on the clients.


This blueprint sounds like its still being fleshed out, so I am 
wondering what the value is of the current patches 
https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z


Those patches mainly sync cliutils and apiutils from oslo into the 
assorted clients. But if this blueprint is about the python API and 
not the CLI (as that would be the openstack-pythonclient), why sync 
in apiutils?


Also does this need to go through oslo-incubator or can this start 
out as a library? Making this a library earlier on will reduce the 
number of patches needed to get 20+ repositories to use this.




Alexei and others have at least started the first stage of a rollout - 
the blueprint(s) needs additional work, planning and discussion, but 
his work is a good first step (reduce the duplication of code) 
although I am worried that the libraries and APIs / namespaces will 
need to change if we continue these discussions which potentially 
means re-doing work.


If we take a step back, a rollout process might be:

1: Solidify the libraries / layout / naming conventions (blueprint)
2: Solidify the APIs exposed to consumers (blueprint)
3: Pick up on the common-client-library-2 work which is primarily a 
migration of common code into oslo today, into the structure defined 
by 1 & 2


So, I sort of agree: moving / collapsing code now might be premature. 
I do strongly agree it should stand on its own as a library rather 
than an oslo incubator however. We should start with a single, clean 
namespace / library rather than depending on oslo directly.
Knowing usual openstack workflow I'm afraid that #1,#2 with a waterfall 
approach may take years to be complete.
And after they'll be approved it will become clear that this 
architecture is already outdated.

We try to use iterative approach for clients refactoring.
We started our work from removing code duplication becau

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 9:26 AM, Justin Hammond 
mailto:justin.hamm...@rackspace.com>> wrote:

I'm not sure if it was said, but which httplib using being used (urllib3
maybe?). Also I noticed many people were talking about supporting auth
properly, but are there any intentions to properly support 'noauth'
(python-neutronclient, for instance, doesn't support it properly as of
this writing)?


Can you detail out noauth for me; and I would say the defacto httplib in python 
today is python-requests - urllib3 is also good but I would say from a 
*consumer* standpoint requests offers more in terms of usability / extensibility

On 1/15/14 10:53 PM, "Alexei Kornienko" 
mailto:alexei.kornie...@gmail.com>> wrote:

I did notice, however, that neutronclient is
conspicuously absent from the Work Items in the blueprint's Whiteboard.

It will surely be added later. We already working on several things in
parallel and we will add neutronclient soon.

Do you need another person to make neutron client?


I would love to see a bit more detail on the structure of the
lib(s), the blueprint really doesn't discuss the
design/organization/intended API of the libs.  For example, I would hope
the distinction between the various layers of a client stack don't get
lost, i.e. not mixing the low-level REST API bits with the higher-level
CLI parsers and decorators.
Does the long-term goals include a common caching layer?


Distinction between client layers won't get lost and would only be
improved. My basic idea is the following:

1) Transport layer would handle all transport related stuff - HTTP, JSON
encoding, auth, caching, etc.

2) Model layer (Resource classes, BaseManager, etc.) will handle data
representation, validation

3) API layer will handle all project specific stuff - url mapping, etc.
(This will be imported to use client in other applications)

4) Cli level will handle all stuff related to cli mapping - argparse,
argcomplete, etc.


I believe the current effort referenced by the blueprint is focusing on
moving existing code into the incubator for reuse, to make it easier to
restructure later. Alexei, do I have that correct?

You are right. The first thing we do is try to make all clients look/work
in similar way. After we'll continue our work with improving overall
structure.






2014/1/16 Noorul Islam K M mailto:noo...@noorul.com>>

Doug Hellmann mailto:doug.hellm...@dreamhost.com>> 
writes:

Several people have mentioned to me that they are interested in, or
actively working on, code related to a "common" client library --
something
meant to be reused directly as a basis for creating a common library for
all of the openstack clients to use. There's a blueprint [1] in oslo,
and I
believe the keystone devs and unified CLI teams are probably interested
in
ensuring that the resulting API ends up meeting all of our various
requirements.

If you're interested in this effort, please subscribe to the blueprint
and
use that to coordinate efforts so we don't produce more than one common
library. ;-)



Solum is already using it https://review.openstack.org/#/c/58067/

I would love to watch this space.

Regards,
Noorul

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev







___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 9:07 AM, Joe Gordon 
mailto:joe.gord...@gmail.com>> wrote:




On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller 
mailto:jesse.nol...@rackspace.com>> wrote:

On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah 
mailto:chmo...@enovance.com>> wrote:


On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones 
mailto:c...@tenshu.net>> wrote:
Once a common library is in place, is there any intention to (or resistance 
against) collapsing the clients into a single project or even a single command 
(a la busybox)?


that's what openstackclient is here for 
https://github.com/openstack/python-openstackclient

After speaking with people working on OSC and looking at the code base in 
depth; I don’t think this addresses what Chris is implying: OSC wraps the 
individual CLIs built by each project today, instead of the inverse: a common 
backend that the individual CLIs can wrap - the latter is an important 
distinction as currently, building a single binary install of OSC for say, 
Windows is difficult given the dependency tree incurred by each of the wrapped 
CLIs, difference in dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client classes / functions / 
methods means that the layer that presents a consistent user interface (OSC) to 
the user is made more complex juggling names/renames/commands/etc.

In the inverted case of what we have today (single backend); as a developer of 
user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(…)

def my_command(cli):
compute = Compute(Auth(cli.tentant…, connect=True))
compute.list_flavors()

This would mean that *even if the individual clients needed or wanted to keep 
their specific CLIs, they would be able to use a not “least common denominator” 
back end (each service can have a rich 
common.api.compute.py or api.compute/client.py 
and extend where needed. However tools like horizon / openstackclient can 
choose not to leverage the “power user/operator/admin” components and present a 
simplified user interface.

I’m working on a wiki page + blueprint to brainstorm how we could accomplish 
this based off of what work is in flight today (see doug’s linked blueprint) 
and sussing out a layout / API strawman for discussion.

Some of the additions that came out of this email threads and others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and each sub-project 
delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing

I am happy to see this work being done, there is definitely a lot of work to be 
done on the clients.

This blueprint sounds like its still being fleshed out, so I am wondering what 
the value is of the current patches 
https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z

Those patches mainly sync cliutils and apiutils from oslo into the assorted 
clients. But if this blueprint is about the python API and not the CLI (as that 
would be the openstack-pythonclient), why sync in apiutils?

Also does this need to go through oslo-incubator or can this start out as a 
library? Making this a library earlier on will reduce the number of patches 
needed to get 20+ repositories to use this.


Alexei and others have at least started the first stage of a rollout - the 
blueprint(s) needs additional work, planning and discussion, but his work is a 
good first step (reduce the duplication of code) although I am worried that the 
libraries and APIs / namespaces will need to change if we continue these 
discussions which potentially means re-doing work.

If we take a step back, a rollout process might be:

1: Solidify the libraries / layout / naming conventions (blueprint)
2: Solidify the APIs exposed to consumers (blueprint)
3: Pick up on the common-client-library-2 work which is primarily a migration 
of common code into oslo today, into the structure defined by 1 & 2

So, I sort of agree: moving / collapsing code now might be premature. I do 
strongly agree it should stand on its own as a library rather than an oslo 
incubator however. We should start with a single, clean namespace / library 
rather than depending on oslo directly.

jesse


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Joe Gordon]

On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller wrote:

>
>  On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah 
> wrote:
>
>
> On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones  wrote:
>
>> Once a common library is in place, is there any intention to (or
>> resistance against) collapsing the clients into a single project or even a
>> single command (a la busybox)?
>
>
>
>  that's what openstackclient is here for
> https://github.com/openstack/python-openstackclient
>
>
>  After speaking with people working on OSC and looking at the code base
> in depth; I don’t think this addresses what Chris is implying: OSC wraps
> the individual CLIs built by each project today, instead of the inverse: a
> common backend that the individual CLIs can wrap - the latter is an
> important distinction as currently, building a single binary install of OSC
> for say, Windows is difficult given the dependency tree incurred by each of
> the wrapped CLIs, difference in dependencies, structure, etc.
>
>  Also, wrapping a series of inconsistent back end Client classes /
> functions / methods means that the layer that presents a consistent user
> interface (OSC) to the user is made more complex juggling
> names/renames/commands/etc.
>
>  In the inverted case of what we have today (single backend); as a
> developer of user interfaces (CLIs, Applications, Web apps (horizon)) you
> would be able to:
>
>  from openstack.common.api import Auth
>  from openstack.common.api import Compute
> from openstack.common.util import cli_tools
>
>  my_cli = cli_tools.build(…)
>
>  def my_command(cli):
> compute = Compute(Auth(cli.tentant…, connect=True))
> compute.list_flavors()
>
>  This would mean that *even if the individual clients needed or wanted to
> keep their specific CLIs, they would be able to use a not “least common
> denominator” back end (each service can have a rich common.api.compute.pyor 
> api.compute/client.py and extend where needed. However tools like
> horizon / openstackclient can choose not to leverage the “power
> user/operator/admin” components and present a simplified user interface.
>
>  I’m working on a wiki page + blueprint to brainstorm how we could
> accomplish this based off of what work is in flight today (see doug’s
> linked blueprint) and sussing out a layout / API strawman for discussion.
>
>  Some of the additions that came out of this email threads and others:
>
>  1. Common backend should provide / offer caching utilities
> 2. Auth retries need to be handled by the auth object, and each
> sub-project delegates to the auth object to manage that.
> 3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing
>

I am happy to see this work being done, there is definitely a lot of work
to be done on the clients.

This blueprint sounds like its still being fleshed out, so I am wondering
what the value is of the current patches
https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z

Those patches mainly sync cliutils and apiutils from oslo into the assorted
clients. But if this blueprint is about the python API and not the CLI (as
that would be the openstack-pythonclient), why sync in apiutils?

Also does this need to go through oslo-incubator or can this start out as a
library? Making this a library earlier on will reduce the number of patches
needed to get 20+ repositories to use this.




>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah 
mailto:chmo...@enovance.com>> wrote:


On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones 
mailto:c...@tenshu.net>> wrote:
Once a common library is in place, is there any intention to (or resistance 
against) collapsing the clients into a single project or even a single command 
(a la busybox)?


that's what openstackclient is here for 
https://github.com/openstack/python-openstackclient

After speaking with people working on OSC and looking at the code base in 
depth; I don’t think this addresses what Chris is implying: OSC wraps the 
individual CLIs built by each project today, instead of the inverse: a common 
backend that the individual CLIs can wrap - the latter is an important 
distinction as currently, building a single binary install of OSC for say, 
Windows is difficult given the dependency tree incurred by each of the wrapped 
CLIs, difference in dependencies, structure, etc.

Also, wrapping a series of inconsistent back end Client classes / functions / 
methods means that the layer that presents a consistent user interface (OSC) to 
the user is made more complex juggling names/renames/commands/etc.

In the inverted case of what we have today (single backend); as a developer of 
user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to:

from openstack.common.api import Auth
from openstack.common.api import Compute
from openstack.common.util import cli_tools

my_cli = cli_tools.build(…)

def my_command(cli):
compute = Compute(Auth(cli.tentant…, connect=True))
compute.list_flavors()

This would mean that *even if the individual clients needed or wanted to keep 
their specific CLIs, they would be able to use a not “least common denominator” 
back end (each service can have a rich common.api.compute.py or 
api.compute/client.py and extend where needed. However tools like horizon / 
openstackclient can choose not to leverage the “power user/operator/admin” 
components and present a simplified user interface.

I’m working on a wiki page + blueprint to brainstorm how we could accomplish 
this based off of what work is in flight today (see doug’s linked blueprint) 
and sussing out a layout / API strawman for discussion.

Some of the additions that came out of this email threads and others:

1. Common backend should provide / offer caching utilities
2. Auth retries need to be handled by the auth object, and each sub-project 
delegates to the auth object to manage that.
3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Chmouel Boudjnah]

On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones  wrote:

> Once a common library is in place, is there any intention to (or
> resistance against) collapsing the clients into a single project or even a
> single command (a la busybox)?



that's what openstackclient is here for
https://github.com/openstack/python-openstackclient
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 16, 2014, at 5:42 AM, "Chris Jones" 
mailto:c...@tenshu.net>> wrote:

Hi

Once a common library is in place, is there any intention to (or resistance 
against) collapsing the clients into a single project or even a single command 
(a la busybox)?

(I'm thinking reduced load for packagers, simpler installation for users, etc)

Cheers,
--
Chris Jones

Based on the email I sent; ideally we'd expand the blueprint to include these 
details and speak to the PTLs for each project.

Ideally (IMO) yes - long term they would be collapsed into one or two packages 
to install. However at very least the code can be made so that the core 
functionality is in the common/single sdk backend and openstackcli, and each 
project CLI can derive their named/branded project specific one for that.

For standalone installs of nova or swift for example; not pushing users to use 
the entire openstackclient makes sense, instead guiding users to use a tool 
dedicated to that.

There are pros and cons on both sides, meeting in the middle at least collapses 
the heavy lifting into the common code base.


On 15 Jan 2014, at 19:37, Doug Hellmann 
mailto:doug.hellm...@dreamhost.com>> wrote:

Several people have mentioned to me that they are interested in, or actively 
working on, code related to a "common" client library -- something meant to be 
reused directly as a basis for creating a common library for all of the 
openstack clients to use. There's a blueprint [1] in oslo, and I believe the 
keystone devs and unified CLI teams are probably interested in ensuring that 
the resulting API ends up meeting all of our various requirements.

If you're interested in this effort, please subscribe to the blueprint and use 
that to coordinate efforts so we don't produce more than one common library. ;-)

Thanks,
Doug


[1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

> On Jan 16, 2014, at 2:09 AM, "Flavio Percoco"  wrote:
> 
>> On 15/01/14 21:35 +, Jesse Noller wrote:
>> 
>>> On Jan 15, 2014, at 1:37 PM, Doug Hellmann  
>>> wrote:
>>> 
>>> Several people have mentioned to me that they are interested in, or 
>>> actively working on, code related to a "common" client library -- something 
>>> meant to be reused directly as a basis for creating a common library for 
>>> all of the openstack clients to use. There's a blueprint [1] in oslo, and I 
>>> believe the keystone devs and unified CLI teams are probably interested in 
>>> ensuring that the resulting API ends up meeting all of our various 
>>> requirements.
>>> 
>>> If you're interested in this effort, please subscribe to the blueprint and 
>>> use that to coordinate efforts so we don't produce more than one common 
>>> library. ;-)
>>> 
>>> Thanks,
>>> Doug
>>> 
>>> 
>>> [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
>> 
>> *raises hand*
>> 
>> Me me!
>> 
>> I’ve been talking to many contributors about the Developer Experience stuff 
>> I emailed out prior to the holidays and I was starting blueprint work, but 
>> this is a great pointer. I’m going to have to sync up with Alexei.
>> 
>> I think solving this for openstack developers and maintainers as the 
>> blueprint says is a big win in terms of code reuse / maintenance and 
>> consistent but more so for *end-user developers* consuming openstack clouds.
>> 
>> Some background - there’s some terminology mismatch but the rough idea is 
>> the same:
>> 
>> * A centralized “SDK” (Software Development Kit) would be built condensing 
>> the common code and logic and operations into a single namespace.
>> 
>> * This SDK would be able to be used by “downstream” CLIs - essentially the 
>> CLIs become a specialized front end - and in some cases, only an argparse or 
>> cliff front-end to the SDK methods located in the (for example) 
>> openstack.client.api.compute
>> 
>> * The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived 
>> clients - all of the openstack.client.api.** classes would accept an Auth 
>> object to delegate management / mocking of the Auth / service catalog stuff 
>> to. This means developers building applications (say for example, horizon) 
>> don’t need to worry about token/expired authentication/etc.
>> 
>> * Simplify the dependency graph & code for the existing tools to enable 
>> single binary installs (py2exe, py2app, etc) for end users of the command 
>> line tools.
>> 
>> Short version: if a developer wants to consume an openstack cloud; the would 
>> have a single SDK with minimal dependencies and import from a single 
>> namespace. An example application might look like:
>> 
>> from openstack.api import AuthV2
>> from openstack.api import ComputeV2
>> 
>> myauth = AuthV2(…., connect=True)
>> compute = ComputeV2(myauth)
>> 
>> compute.list_flavors()
> 
> I know this is an example but, could we leave the version out of the
> class name? Having something like:
> 
> from openstack.api.v2 import Compute
> 
>   or
> 
> from openstack.compute.v2 import Instance
> 
> (just made that up)
> 
> for marconi we're using the later.

Definitely; it should be based on namespaces. 

> 
>> This greatly improves the developer experience both internal to openstack 
>> and externally. Currently OpenStack has 22+ (counting stackforge) potential 
>> libraries a developer may need to install to use a full deployment of 
>> OpenStack:
>> 
>> * python-keystoneclient (identity)
>> * python-glanceclient (image)
>> * python-novaclient (compute)
>> * python-troveclient (database)
>> * python-neutronclient (network)
>> * python-ironicclient (bare metal)
>> * python-heatclient (orchestration)
>> * python-cinderclient (block storage)
>> * python-ceilometerclient (telemetry, metrics & billing)
>> * python-swiftclient (object storage)
>> * python-savannaclient (big data)
>> * python-openstackclient (meta client package)
>> * python-marconiclient (queueing)
>> * python-tuskarclient (tripleo / management)
>> * python-melangeclient (dead)
>> * python-barbicanclient (secrets)
>> * python-solumclient (ALM)
>> * python-muranoclient (application catalog)
>> * python-manilaclient (shared filesystems)
>> * python-libraclient (load balancers)
>> * python-climateclient (reservations)
>> * python-designateclient (Moniker/DNS)
>> 
>> If you exclude the above and look on PyPI:
>> 
>> On PyPi (client libraries/SDKs only, excluding the above - not maintained by 
>> openstack):
>> 
>> * hpcloud-auth-openstack 1.0
>> * python-openstackclient 0.2.2
>> * rackspace-auth-openstack 1.1
>> * posthaste 0.2.2
>> * pyrax 1.6.2
>> * serverherald 0.0.1
>> * warm 0.3.1
>> * vaporize 0.3.2
>> * swiftsc (https://github.com/mkouhei/swiftsc)
>> * bookofnova 0.007
>> * nova-adminclient 0.1.8
>> * python-quantumclient 2.2.4.3
>> * python-stackhelper 0.0.7.1.gcab1eb0
>> * swift-bench 1.0
>> * swiftly 1.12
>> * txAWS 0.2.3
>> * cfupload 0.5.1
>> * python-reddwarfcli

Re: [openstack-dev] a "common" client library

[Chris Jones]

Hi

Once a common library is in place, is there any intention to (or resistance 
against) collapsing the clients into a single project or even a single command 
(a la busybox)?

(I'm thinking reduced load for packagers, simpler installation for users, etc)

Cheers,
--
Chris Jones

> On 15 Jan 2014, at 19:37, Doug Hellmann  wrote:
> 
> Several people have mentioned to me that they are interested in, or actively 
> working on, code related to a "common" client library -- something meant to 
> be reused directly as a basis for creating a common library for all of the 
> openstack clients to use. There's a blueprint [1] in oslo, and I believe the 
> keystone devs and unified CLI teams are probably interested in ensuring that 
> the resulting API ends up meeting all of our various requirements.
> 
> If you're interested in this effort, please subscribe to the blueprint and 
> use that to coordinate efforts so we don't produce more than one common 
> library. ;-)
> 
> Thanks,
> Doug
> 
> 
> [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Flavio Percoco]

On 15/01/14 21:35 +, Jesse Noller wrote:


On Jan 15, 2014, at 1:37 PM, Doug Hellmann  wrote:


Several people have mentioned to me that they are interested in, or actively working on, 
code related to a "common" client library -- something meant to be reused 
directly as a basis for creating a common library for all of the openstack clients to 
use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI 
teams are probably interested in ensuring that the resulting API ends up meeting all of 
our various requirements.

If you're interested in this effort, please subscribe to the blueprint and use 
that to coordinate efforts so we don't produce more than one common library. ;-)

Thanks,
Doug


[1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2


*raises hand*

Me me!

I’ve been talking to many contributors about the Developer Experience stuff I 
emailed out prior to the holidays and I was starting blueprint work, but this 
is a great pointer. I’m going to have to sync up with Alexei.

I think solving this for openstack developers and maintainers as the blueprint 
says is a big win in terms of code reuse / maintenance and consistent but more 
so for *end-user developers* consuming openstack clouds.

Some background - there’s some terminology mismatch but the rough idea is the 
same:

* A centralized “SDK” (Software Development Kit) would be built condensing the 
common code and logic and operations into a single namespace.

* This SDK would be able to be used by “downstream” CLIs - essentially the CLIs 
become a specialized front end - and in some cases, only an argparse or cliff 
front-end to the SDK methods located in the (for example) 
openstack.client.api.compute

* The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived 
clients - all of the openstack.client.api.** classes would accept an Auth 
object to delegate management / mocking of the Auth / service catalog stuff to. 
This means developers building applications (say for example, horizon) don’t 
need to worry about token/expired authentication/etc.

* Simplify the dependency graph & code for the existing tools to enable single 
binary installs (py2exe, py2app, etc) for end users of the command line tools.

Short version: if a developer wants to consume an openstack cloud; the would 
have a single SDK with minimal dependencies and import from a single namespace. 
An example application might look like:

from openstack.api import AuthV2
from openstack.api import ComputeV2

myauth = AuthV2(…., connect=True)
compute = ComputeV2(myauth)

compute.list_flavors()



I know this is an example but, could we leave the version out of the
class name? Having something like:

from openstack.api.v2 import Compute

   or

from openstack.compute.v2 import Instance

(just made that up)

for marconi we're using the later.


This greatly improves the developer experience both internal to openstack and 
externally. Currently OpenStack has 22+ (counting stackforge) potential 
libraries a developer may need to install to use a full deployment of OpenStack:

 * python-keystoneclient (identity)
 * python-glanceclient (image)
 * python-novaclient (compute)
 * python-troveclient (database)
 * python-neutronclient (network)
 * python-ironicclient (bare metal)
 * python-heatclient (orchestration)
 * python-cinderclient (block storage)
 * python-ceilometerclient (telemetry, metrics & billing)
 * python-swiftclient (object storage)
 * python-savannaclient (big data)
 * python-openstackclient (meta client package)
 * python-marconiclient (queueing)
 * python-tuskarclient (tripleo / management)
 * python-melangeclient (dead)
 * python-barbicanclient (secrets)
 * python-solumclient (ALM)
 * python-muranoclient (application catalog)
 * python-manilaclient (shared filesystems)
 * python-libraclient (load balancers)
 * python-climateclient (reservations)
 * python-designateclient (Moniker/DNS)

If you exclude the above and look on PyPI:

On PyPi (client libraries/SDKs only, excluding the above - not maintained by 
openstack):

* hpcloud-auth-openstack 1.0
* python-openstackclient 0.2.2
* rackspace-auth-openstack 1.1
* posthaste 0.2.2
* pyrax 1.6.2
* serverherald 0.0.1
* warm 0.3.1
* vaporize 0.3.2
* swiftsc (https://github.com/mkouhei/swiftsc)
* bookofnova 0.007
* nova-adminclient 0.1.8
* python-quantumclient 2.2.4.3
* python-stackhelper 0.0.7.1.gcab1eb0
* swift-bench 1.0
* swiftly 1.12
* txAWS 0.2.3
* cfupload 0.5.1
* python-reddwarfclient 0.1.2
* python-automationclient 1.2.1
* rackspace-glanceclient 0.9
* rackspace-novaclient 1.4

If you ignore PyPI and just want to install the base say - 7 services, each one 
of the python-** clients has its own dependency tree and may or may not build 
from one of the others. If a vendor wants to extend any of them, it’s basically 
a fork instead of a clean plugin system.

On the CLI front - this would *greatly* simplify the work openstackclient needs 
to do - it would be able

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 15, 2014, at 6:16 PM, Doug Hellmann  wrote:

> 
> 
> 
> On Wed, Jan 15, 2014 at 5:39 PM, Dean Troyer  wrote:
> On Wed, Jan 15, 2014 at 1:37 PM, Doug Hellmann  
> wrote:
> Several people have mentioned to me that they are interested in, or actively 
> working on, code related to a "common" client library -- something meant to 
> be reused directly as a basis for creating a common library for all of the 
> openstack clients to use. There's a blueprint [1] in oslo, and I believe the 
> keystone devs and unified CLI teams are probably interested in ensuring that 
> the resulting API ends up meeting all of our various requirements.
> 
> I would love to see a bit more detail on the structure of the lib(s), the 
> blueprint really doesn't discuss the design/organization/intended API of the 
> libs.  For example, I would hope the distinction between the various layers 
> of a client stack don't get lost, i.e. not mixing the low-level REST API bits 
> with the higher-level CLI parsers and decorators.
> 
> Does the long-term goals include a common caching layer?
> 
> Good questions. I believe the current effort referenced by the blueprint is 
> focusing on moving existing code into the incubator for reuse, to make it 
> easier to restructure later. Alexei, do I have that correct?
> 
> I was starting to feel like an (unnecessary) bottleneck, so my goal for this 
> email thread was to step out of the middle of the conversation, and make sure 
> everyone who is doing the work knows about each other so you can all 
> coordinate directly. 
> 
> If we need more details for requirements and a design, let's start writing 
> them up. Maybe a wiki page or etherpad to get started, and turn that into 
> blueprints when ready? Who wants to take that on?

I’ll take that on: If I schedule an IRC meeting next week; who would like to 
join?

> 
> Doug
> 
>  
> 
> dt
> 
> -- 
> 
> Dean Troyer
> dtro...@gmail.com
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Alexei Kornienko]

>I did notice, however, that neutronclient is
conspicuously absent from the Work Items in the blueprint's Whiteboard.
It will surely be added later. We already working on several things in
parallel and we will add neutronclient soon.

>I would love to see a bit more detail on the structure of the lib(s), the
blueprint really doesn't discuss the design/organization/intended API of
the libs.  For example, I would hope the distinction between the various
layers of a client stack don't get lost, i.e. not mixing the low-level REST
API bits with the higher-level CLI parsers and decorators.
>Does the long-term goals include a common caching layer?

Distinction between client layers won't get lost and would only be
improved. My basic idea is the following:
1) Transport layer would handle all transport related stuff - HTTP, JSON
encoding, auth, caching, etc.
2) Model layer (Resource classes, BaseManager, etc.) will handle data
representation, validation
3) API layer will handle all project specific stuff - url mapping, etc.
(This will be imported to use client in other applications)
4) Cli level will handle all stuff related to cli mapping - argparse,
argcomplete, etc.

>I believe the current effort referenced by the blueprint is focusing on
moving existing code into the incubator for reuse, to make it easier to
restructure later. Alexei, do I have that correct?
You are right. The first thing we do is try to make all clients look/work
in similar way. After we'll continue our work with improving overall
structure.




2014/1/16 Noorul Islam K M 

> Doug Hellmann  writes:
>
> > Several people have mentioned to me that they are interested in, or
> > actively working on, code related to a "common" client library --
> something
> > meant to be reused directly as a basis for creating a common library for
> > all of the openstack clients to use. There's a blueprint [1] in oslo,
> and I
> > believe the keystone devs and unified CLI teams are probably interested
> in
> > ensuring that the resulting API ends up meeting all of our various
> > requirements.
> >
> > If you're interested in this effort, please subscribe to the blueprint
> and
> > use that to coordinate efforts so we don't produce more than one common
> > library. ;-)
> >
>
> Solum is already using it https://review.openstack.org/#/c/58067/
>
> I would love to watch this space.
>
> Regards,
> Noorul
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Noorul Islam K M]

Doug Hellmann  writes:

> Several people have mentioned to me that they are interested in, or
> actively working on, code related to a "common" client library -- something
> meant to be reused directly as a basis for creating a common library for
> all of the openstack clients to use. There's a blueprint [1] in oslo, and I
> believe the keystone devs and unified CLI teams are probably interested in
> ensuring that the resulting API ends up meeting all of our various
> requirements.
>
> If you're interested in this effort, please subscribe to the blueprint and
> use that to coordinate efforts so we don't produce more than one common
> library. ;-)
>

Solum is already using it https://review.openstack.org/#/c/58067/

I would love to watch this space.

Regards,
Noorul

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Doug Hellmann]

On Wed, Jan 15, 2014 at 5:39 PM, Dean Troyer  wrote:

> On Wed, Jan 15, 2014 at 1:37 PM, Doug Hellmann <
> doug.hellm...@dreamhost.com> wrote:
>
>> Several people have mentioned to me that they are interested in, or
>> actively working on, code related to a "common" client library -- something
>> meant to be reused directly as a basis for creating a common library for
>> all of the openstack clients to use. There's a blueprint [1] in oslo, and I
>> believe the keystone devs and unified CLI teams are probably interested in
>> ensuring that the resulting API ends up meeting all of our various
>> requirements.
>>
>
> I would love to see a bit more detail on the structure of the lib(s), the
> blueprint really doesn't discuss the design/organization/intended API of
> the libs.  For example, I would hope the distinction between the various
> layers of a client stack don't get lost, i.e. not mixing the low-level REST
> API bits with the higher-level CLI parsers and decorators.
>
> Does the long-term goals include a common caching layer?
>

Good questions. I believe the current effort referenced by the blueprint
is focusing on moving existing code into the incubator for reuse, to make
it easier to restructure later. Alexei, do I have that correct?

I was starting to feel like an (unnecessary) bottleneck, so my goal for
this email thread was to step out of the middle of the conversation, and
make sure everyone who is doing the work knows about each other so you can
all coordinate directly.

If we need more details for requirements and a design, let's start writing
them up. Maybe a wiki page or etherpad to get started, and turn that into
blueprints when ready? Who wants to take that on?

Doug



>
> dt
>
> --
>
> Dean Troyer
> dtro...@gmail.com
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 15, 2014, at 4:55 PM, "Renat Akhmerov" 
mailto:rakhme...@mirantis.com>> wrote:

Great idea, fully support it. We’re interested in that too. One specific thing 
that was mentioned is the ability to mock auth service seems to be very useful 
for some test scenarios, we came across that recently.

Renat Akhmerov
@ Mirantis Inc.

I'd add that verified fakes/mocks/stubs for all network traffic calls are 
critical for end users wanting to build/test code against the common code/sdk 
so they don't need a fully running OpenStack install that incurs charges. For 
fast unit test it's critical



On 15 Jan 2014, at 14:07, Sylvain Bauza 
mailto:sylvain.ba...@gmail.com>> wrote:

Hi Doug,

Count me in. Climate is currently working on delivering its first 
python-climateclient but it would be great if we could leverage any olso lib 
for this.

-Sylvain


2014/1/15 Doug Hellmann 
mailto:doug.hellm...@dreamhost.com>>
Several people have mentioned to me that they are interested in, or actively 
working on, code related to a "common" client library -- something meant to be 
reused directly as a basis for creating a common library for all of the 
openstack clients to use. There's a blueprint [1] in oslo, and I believe the 
keystone devs and unified CLI teams are probably interested in ensuring that 
the resulting API ends up meeting all of our various requirements.

If you're interested in this effort, please subscribe to the blueprint and use 
that to coordinate efforts so we don't produce more than one common library. ;-)

Thanks,
Doug


[1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Renat Akhmerov]

Great idea, fully support it. We’re interested in that too. One specific thing 
that was mentioned is the ability to mock auth service seems to be very useful 
for some test scenarios, we came across that recently.

Renat Akhmerov
@ Mirantis Inc.

On 15 Jan 2014, at 14:07, Sylvain Bauza  wrote:

> Hi Doug,
> 
> Count me in. Climate is currently working on delivering its first 
> python-climateclient but it would be great if we could leverage any olso lib 
> for this.
> 
> -Sylvain
> 
> 
> 2014/1/15 Doug Hellmann 
> Several people have mentioned to me that they are interested in, or actively 
> working on, code related to a "common" client library -- something meant to 
> be reused directly as a basis for creating a common library for all of the 
> openstack clients to use. There's a blueprint [1] in oslo, and I believe the 
> keystone devs and unified CLI teams are probably interested in ensuring that 
> the resulting API ends up meeting all of our various requirements.
> 
> If you're interested in this effort, please subscribe to the blueprint and 
> use that to coordinate efforts so we don't produce more than one common 
> library. ;-)
> 
> Thanks,
> Doug
> 
> 
> [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Dean Troyer]

On Wed, Jan 15, 2014 at 1:37 PM, Doug Hellmann
wrote:

> Several people have mentioned to me that they are interested in, or
> actively working on, code related to a "common" client library -- something
> meant to be reused directly as a basis for creating a common library for
> all of the openstack clients to use. There's a blueprint [1] in oslo, and I
> believe the keystone devs and unified CLI teams are probably interested in
> ensuring that the resulting API ends up meeting all of our various
> requirements.
>

I would love to see a bit more detail on the structure of the lib(s), the
blueprint really doesn't discuss the design/organization/intended API of
the libs.  For example, I would hope the distinction between the various
layers of a client stack don't get lost, i.e. not mixing the low-level REST
API bits with the higher-level CLI parsers and decorators.

Does the long-term goals include a common caching layer?

dt

-- 

Dean Troyer
dtro...@gmail.com
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Sylvain Bauza]

Hi Doug,

Count me in. Climate is currently working on delivering its first
python-climateclient but it would be great if we could leverage any olso
lib for this.

-Sylvain


2014/1/15 Doug Hellmann 

> Several people have mentioned to me that they are interested in, or
> actively working on, code related to a "common" client library -- something
> meant to be reused directly as a basis for creating a common library for
> all of the openstack clients to use. There's a blueprint [1] in oslo, and I
> believe the keystone devs and unified CLI teams are probably interested in
> ensuring that the resulting API ends up meeting all of our various
> requirements.
>
> If you're interested in this effort, please subscribe to the blueprint and
> use that to coordinate efforts so we don't produce more than one common
> library. ;-)
>
> Thanks,
> Doug
>
>
> [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Doug Hellmann]

On Wed, Jan 15, 2014 at 4:45 PM, Jay Pipes  wrote:

> On Wed, 2014-01-15 at 14:37 -0500, Doug Hellmann wrote:
> > Several people have mentioned to me that they are interested in, or
> > actively working on, code related to a "common" client library --
> > something meant to be reused directly as a basis for creating a common
> > library for all of the openstack clients to use. There's a blueprint
> > [1] in oslo, and I believe the keystone devs and unified CLI teams are
> > probably interested in ensuring that the resulting API ends up meeting
> > all of our various requirements.
> >
> > If you're interested in this effort, please subscribe to the blueprint
> > and use that to coordinate efforts so we don't produce more than one
> > common library. ;-)
>
> ++ on the effort. I did notice, however, that neutronclient is
> conspicuously absent from the Work Items in the blueprint's Whiteboard.
> Is there a reason for this, or just a mistaken omission?
>

Great question.

Alexei, have you talked to Mark McClain about participating in this?

Doug



>
> Best,
> -jay
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jay Pipes]

On Wed, 2014-01-15 at 14:37 -0500, Doug Hellmann wrote:
> Several people have mentioned to me that they are interested in, or
> actively working on, code related to a "common" client library --
> something meant to be reused directly as a basis for creating a common
> library for all of the openstack clients to use. There's a blueprint
> [1] in oslo, and I believe the keystone devs and unified CLI teams are
> probably interested in ensuring that the resulting API ends up meeting
> all of our various requirements.
> 
> If you're interested in this effort, please subscribe to the blueprint
> and use that to coordinate efforts so we don't produce more than one
> common library. ;-)

++ on the effort. I did notice, however, that neutronclient is
conspicuously absent from the Work Items in the blueprint's Whiteboard.
Is there a reason for this, or just a mistaken omission?

Best,
-jay



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] a "common" client library

[Jesse Noller]

On Jan 15, 2014, at 1:37 PM, Doug Hellmann  wrote:

> Several people have mentioned to me that they are interested in, or actively 
> working on, code related to a "common" client library -- something meant to 
> be reused directly as a basis for creating a common library for all of the 
> openstack clients to use. There's a blueprint [1] in oslo, and I believe the 
> keystone devs and unified CLI teams are probably interested in ensuring that 
> the resulting API ends up meeting all of our various requirements.
> 
> If you're interested in this effort, please subscribe to the blueprint and 
> use that to coordinate efforts so we don't produce more than one common 
> library. ;-)
> 
> Thanks,
> Doug
> 
> 
> [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2

*raises hand* 

Me me! 

I’ve been talking to many contributors about the Developer Experience stuff I 
emailed out prior to the holidays and I was starting blueprint work, but this 
is a great pointer. I’m going to have to sync up with Alexei.

I think solving this for openstack developers and maintainers as the blueprint 
says is a big win in terms of code reuse / maintenance and consistent but more 
so for *end-user developers* consuming openstack clouds.

Some background - there’s some terminology mismatch but the rough idea is the 
same:

* A centralized “SDK” (Software Development Kit) would be built condensing the 
common code and logic and operations into a single namespace.

* This SDK would be able to be used by “downstream” CLIs - essentially the CLIs 
become a specialized front end - and in some cases, only an argparse or cliff 
front-end to the SDK methods located in the (for example) 
openstack.client.api.compute 

* The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived 
clients - all of the openstack.client.api.** classes would accept an Auth 
object to delegate management / mocking of the Auth / service catalog stuff to. 
This means developers building applications (say for example, horizon) don’t 
need to worry about token/expired authentication/etc. 

* Simplify the dependency graph & code for the existing tools to enable single 
binary installs (py2exe, py2app, etc) for end users of the command line tools.

Short version: if a developer wants to consume an openstack cloud; the would 
have a single SDK with minimal dependencies and import from a single namespace. 
An example application might look like:

from openstack.api import AuthV2
from openstack.api import ComputeV2

myauth = AuthV2(…., connect=True)
compute = ComputeV2(myauth)

compute.list_flavors()

This greatly improves the developer experience both internal to openstack and 
externally. Currently OpenStack has 22+ (counting stackforge) potential 
libraries a developer may need to install to use a full deployment of OpenStack:

  * python-keystoneclient (identity)
  * python-glanceclient (image)
  * python-novaclient (compute)
  * python-troveclient (database)
  * python-neutronclient (network)
  * python-ironicclient (bare metal)
  * python-heatclient (orchestration)
  * python-cinderclient (block storage)
  * python-ceilometerclient (telemetry, metrics & billing)
  * python-swiftclient (object storage)
  * python-savannaclient (big data)
  * python-openstackclient (meta client package)
  * python-marconiclient (queueing)
  * python-tuskarclient (tripleo / management)
  * python-melangeclient (dead)
  * python-barbicanclient (secrets)
  * python-solumclient (ALM)
  * python-muranoclient (application catalog)
  * python-manilaclient (shared filesystems)
  * python-libraclient (load balancers)
  * python-climateclient (reservations)
  * python-designateclient (Moniker/DNS)

If you exclude the above and look on PyPI:

On PyPi (client libraries/SDKs only, excluding the above - not maintained by 
openstack):

 * hpcloud-auth-openstack 1.0
 * python-openstackclient 0.2.2
 * rackspace-auth-openstack 1.1
 * posthaste 0.2.2
 * pyrax 1.6.2
 * serverherald 0.0.1
 * warm 0.3.1
 * vaporize 0.3.2
 * swiftsc (https://github.com/mkouhei/swiftsc)
 * bookofnova 0.007
 * nova-adminclient 0.1.8
 * python-quantumclient 2.2.4.3
 * python-stackhelper 0.0.7.1.gcab1eb0
 * swift-bench 1.0
 * swiftly 1.12
 * txAWS 0.2.3 
 * cfupload 0.5.1
 * python-reddwarfclient 0.1.2
 * python-automationclient 1.2.1
 * rackspace-glanceclient 0.9
 * rackspace-novaclient 1.4

If you ignore PyPI and just want to install the base say - 7 services, each one 
of the python-** clients has its own dependency tree and may or may not build 
from one of the others. If a vendor wants to extend any of them, it’s basically 
a fork instead of a clean plugin system.

On the CLI front - this would *greatly* simplify the work openstackclient needs 
to do - it would be able to import from the main SDK and simply provide the 
noun-verb command line and any other end-user sugar it wanted to. Even if each 
service wanted to keep its own python-X client instead of relying on 
openstackclient it wou

[openstack-dev] a "common" client library

[Doug Hellmann]

Several people have mentioned to me that they are interested in, or
actively working on, code related to a "common" client library -- something
meant to be reused directly as a basis for creating a common library for
all of the openstack clients to use. There's a blueprint [1] in oslo, and I
believe the keystone devs and unified CLI teams are probably interested in
ensuring that the resulting API ends up meeting all of our various
requirements.

If you're interested in this effort, please subscribe to the blueprint and
use that to coordinate efforts so we don't produce more than one common
library. ;-)

Thanks,
Doug


[1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev