Re: [openstack-dev] a common client library
On 17 Jan 2014, at 22:00, Jamie Lennox jamielen...@redhat.com wrote: (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) +1 Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 17 Jan 2014, at 22:06, Robert Collins robe...@robertcollins.net wrote: On 17 January 2014 09:22, Renat Akhmerov rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Has anyone ever considered an idea of generating a fully functional REST client automatically based on an API specification (WADL could be used for that)? Not sure how convenient it would be, it really depends on a particular implementation, but as an idea it could be at least thought of. Sounds a little bit crazy though, I recognize it :). Launchpadlib which builds on wadllib did *exactly* that. It worked fairly well with the one caveat that it fell into the ORM trap - just in time lookups for everything with crippling roundtrips. Thanks, I’ll have a look at it. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 21, 2014, at 10:54 AM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: On 17 Jan 2014, at 22:00, Jamie Lennox jamielen...@redhat.commailto:jamielen...@redhat.com wrote: (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) +1 Renat Akhmerov @ Mirantis Inc. Do you use any other platform than Linux? Even donald - one of the python packaging leads and PyPI leads said this is a bad end-user experience for consumers of openstack clouds. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 18 Jan 2014, at 07:48, Sean Dague s...@dague.net wrote: I also think auto generated clients have a lot of challenges in the same way that full javascript pages in browsers have. If you screw up in a subtle way you can just completely disable your clients from connecting to your server entirely (or block them from using bits of the lesser used function path because a minor bit of schema fat fingered). So without a ton of additional verification on that path, I wouldn't want that anywhere near openstack. Especially with vendor extensions, which mean that enabling a network driver might break all your clients. Yes, this makes tons of sense. The more I think about it the more I come to a conclusion that it will never work well enough (more organizationally though than technically). Additionally, there can be not 1 to 1 match between API methods and python code. For example, we’ve been considering some improvements in Mistral client library so that we can make in some cases a series of calls without actually hitting the server therefore implementing a masking behavior (in some scenarios it seems to make a lot of sense). Typically I’m talking about building a complex object from small pieces. We haven’t made a decision though. We’ll be discussing it with the community. Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 01/21/2014 11:54 AM, Renat Akhmerov wrote: On 17 Jan 2014, at 22:00, Jamie Lennox jamielen...@redhat.com mailto:jamielen...@redhat.com wrote: (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) +1 Renat Akhmerov @ Mirantis Inc. Man, where were you then when we had to spend 3 weeks unwinding global requirements in the gate because pip was figuring it out all kinds of wrong, and we'd do things like uninstall and reinstall python-keystoneclient 6 times during an install. Because after that experience, I'm very anti pip will figure the rest out. Because it won't, not in python, where we're talking about libraries that are in the global namespace, where python can only have 1 version of a dependency installed. If the the solution is every openstack project should install a venv for all it's dependencies to get around this issue, then we're talking a different problem (and a different architecture from what we've been trying to do). But I find the idea of having 12 copies of python-keystone client installed on my openstack environment to be distasteful. So come spend a month working on requirements updates in OpenStack gate... and if you still believe pip will figure it out, well you are a braver man than I. :) -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
Hello, I would like to end this requirements talk cause it doesn't make any sense in term of python clients. Initially the discussion was about many clients projects with separate requirements VS single client project with single requirements list. At that moment we should have stop and actually open requirements lists for python clients. Basically all clients have the same requirement (cause they all do the same stuff - sending HTTP requests K.O.) There is absolutely no difference in the situation of many clients vs single client. Answering another question about user only needs X (keystone) and we install package with clients for all openstack services: Size of keystone client (and any other client I suppose) is ~300Kb I don't think that it's a big difference for the user to install package that is ~300Kb or ~10Mb (unless we are using openstack from Android). From the user perspective I think it's much easier to use client with everything included rather than try to google for client package for some rarely used service. Regards, Alexei 2014/1/21 Sean Dague s...@dague.net On 01/21/2014 11:54 AM, Renat Akhmerov wrote: On 17 Jan 2014, at 22:00, Jamie Lennox jamielen...@redhat.com mailto:jamielen...@redhat.com wrote: (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) +1 Renat Akhmerov @ Mirantis Inc. Man, where were you then when we had to spend 3 weeks unwinding global requirements in the gate because pip was figuring it out all kinds of wrong, and we'd do things like uninstall and reinstall python-keystoneclient 6 times during an install. Because after that experience, I'm very anti pip will figure the rest out. Because it won't, not in python, where we're talking about libraries that are in the global namespace, where python can only have 1 version of a dependency installed. If the the solution is every openstack project should install a venv for all it's dependencies to get around this issue, then we're talking a different problem (and a different architecture from what we've been trying to do). But I find the idea of having 12 copies of python-keystone client installed on my openstack environment to be distasteful. So come spend a month working on requirements updates in OpenStack gate... and if you still believe pip will figure it out, well you are a braver man than I. :) -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
I would like to propose to use this thread to gather and discuss software requirements that our clients should meet. Later we'll summarize all the requirements and use them during our work of improving the clients. By reaching listed requirements we'll be able to evaluate the success of our refactoring. To start this discussion: * API versioning should be encapsulated inside of the client. From the end user perspective using the code should look something like: import openstack.nova nova = openstack.nova.create_client(...params TBD...) nova.start_server(...) Reasoning for this requirement: If user would have to specify version explicitly his code will only work with specific version of client/api or user would have to write several code paths for different api versions. Implementation: We will have an interface (abstact class) that would contain all the methods available in latest API. Different versions of the API would implement this methods. In case method cannot be implemented with existing API it should raise UnsupportedOperation exception. Glossary: user - developer who uses the client/openstack API Please share your thoughts about this requirement/other requirements. Looking forward for your replies, Alexei Kornienko ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 21, 2014, at 12:19 PM, Alexei Kornienko alexei.kornie...@gmail.commailto:alexei.kornie...@gmail.com wrote: Hello, I would like to end this requirements talk cause it doesn't make any sense in term of python clients. Initially the discussion was about many clients projects with separate requirements VS single client project with single requirements list. Then I suspect we’re at an impasse; I do plan on proceeding with a new wiki page and blueprint for a unified client, SDK (backend) for end users. I for one do not think things are as clear cut as you make them - and the +1s to the unified client thoughts clearly express the discontent with the current structure packaging. At that moment we should have stop and actually open requirements lists for python clients. Basically all clients have the same requirement (cause they all do the same stuff - sending HTTP requests K.O.) There is absolutely no difference in the situation of many clients vs single client. Answering another question about user only needs X (keystone) and we install package with clients for all openstack services: Size of keystone client (and any other client I suppose) is ~300Kb I don't think that it's a big difference for the user to install package that is ~300Kb or ~10Mb (unless we are using openstack from Android). It is when most openstack clouds don’t just run keystone. Or nova, or swift. Or when each client acts, smells and behaves differently. It matters a LOT when you’re trying to write applications on top of a mature openstack deployment. From the user perspective I think it's much easier to use client with everything included rather than try to google for client package for some rarely used service. I have actual experience here as a person running a cloud and supporting end-users developers: the overwhelming customer feedback (paid and unpaid (exploratory users)) is that the 22+ clients are confusing, difficult to use, prone to error. There are two ways of resolving this if you’re in my shoes - or in a role where the primary focus is not openstack developers or builders; the first is you coordinate work focusing on developer end user experience upstream, working with openstack and the teams to come up with something that benefits everyone, or, you fork, and build the openstack equivalent to boto / awscli so you can provide a unified “one obvious way to consume openstack clouds”. jesse Regards, Alexei 2014/1/21 Sean Dague s...@dague.netmailto:s...@dague.net On 01/21/2014 11:54 AM, Renat Akhmerov wrote: On 17 Jan 2014, at 22:00, Jamie Lennox jamielen...@redhat.commailto:jamielen...@redhat.com mailto:jamielen...@redhat.commailto:jamielen...@redhat.com wrote: (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) +1 Renat Akhmerov @ Mirantis Inc. Man, where were you then when we had to spend 3 weeks unwinding global requirements in the gate because pip was figuring it out all kinds of wrong, and we'd do things like uninstall and reinstall python-keystoneclient 6 times during an install. Because after that experience, I'm very anti pip will figure the rest out. Because it won't, not in python, where we're talking about libraries that are in the global namespace, where python can only have 1 version of a dependency installed. If the the solution is every openstack project should install a venv for all it's dependencies to get around this issue, then we're talking a different problem (and a different architecture from what we've been trying to do). But I find the idea of having 12 copies of python-keystone client installed on my openstack environment to be distasteful. So come spend a month working on requirements updates in OpenStack gate... and if you still believe pip will figure it out, well you are a braver man than I. :) -Sean -- Sean Dague Samsung Research America s...@dague.netmailto:s...@dague.net / sean.da...@samsung.commailto:sean.da...@samsung.com http://dague.nethttp://dague.net/ ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
don't do this here - the thread is way too deep already. If we get into discussing individual points let's do one question per thread and prefix the emails with [client] or something to tie it all together - Original Message - From: Alexei Kornienko alexei.kornie...@gmail.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Wednesday, 22 January, 2014 4:35:39 AM Subject: Re: [openstack-dev] a common client library I would like to propose to use this thread to gather and discuss software requirements that our clients should meet. Later we'll summarize all the requirements and use them during our work of improving the clients. By reaching listed requirements we'll be able to evaluate the success of our refactoring. To start this discussion: * API versioning should be encapsulated inside of the client. From the end user perspective using the code should look something like: import openstack.nova nova = openstack.nova.create_client(...params TBD...) nova.start_server(...) Reasoning for this requirement: If user would have to specify version explicitly his code will only work with specific version of client/api or user would have to write several code paths for different api versions. Implementation: We will have an interface (abstact class) that would contain all the methods available in latest API. Different versions of the API would implement this methods. In case method cannot be implemented with existing API it should raise UnsupportedOperation exception. Glossary: user - developer who uses the client/openstack API Please share your thoughts about this requirement/other requirements. Looking forward for your replies, Alexei Kornienko ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
It is when most openstack clouds don’t just run keystone. Or nova, or swift. Or when each client acts, smells and behaves differently. It matters a LOT when you’re trying to write applications on top of a mature openstack deployment. I still don't understand the problem here. Installed packages usually just lie quietly on your disk (which size is now measured in terabytes) and they don't act or stink unless you ask them to. I'm pretty sure that most of the people are not aware of ALL packages installed in their distribution and few packages that are not used won't make it any worse I have actual experience here as a person running a cloud and supporting end-users developers: the overwhelming customer feedback (paid and unpaid (exploratory users)) is that the 22+ clients are confusing, difficult to use, prone to error. There are two ways of resolving this if you’re in my shoes - or in a role where the primary focus is not openstack developers or builders; the first is you coordinate work focusing on developer end user experience upstream, working with openstack and the teams to come up with something that benefits everyone, or, you fork, and build the openstack equivalent to boto / awscli so you can provide a unified “one obvious way to consume openstack clouds”. I agree that many clients can be confusing however I think that actuall problem here is not the clients number but discrepancies in client parameters naming and missing/unclear help. In our work of clients refactoring I will also update bash completition that we have for our clients. So when you use nova start-server you will see options applicable to this command etc. For the unified client we can make a nice commands hierarhcy and I don't think it will be confusing to users for example: $ openstack help nova cloud computing fabric controller glance discovering, registering, retrieving and storing virtual machine images cinder ... 22 records $ openstack help nova OpenStack Nova provides a cloud computing fabric controller, supporting a wide variety of virtualization technologies, including KVM, Xen, LXC, VMware, and more. In addition to its native API, it includes compatibility with the commonly encountered Amazon EC2 and S3 APIs. start-server ... With proper bash completition it should be quite easy to use. 2014/1/21 Jesse Noller jesse.nol...@rackspace.com On Jan 21, 2014, at 12:19 PM, Alexei Kornienko alexei.kornie...@gmail.com wrote: Hello, I would like to end this requirements talk cause it doesn't make any sense in term of python clients. Initially the discussion was about many clients projects with separate requirements VS single client project with single requirements list. Then I suspect we’re at an impasse; I do plan on proceeding with a new wiki page and blueprint for a unified client, SDK (backend) for end users. I for one do not think things are as clear cut as you make them - and the +1s to the unified client thoughts clearly express the discontent with the current structure packaging. At that moment we should have stop and actually open requirements lists for python clients. Basically all clients have the same requirement (cause they all do the same stuff - sending HTTP requests K.O.) There is absolutely no difference in the situation of many clients vs single client. Answering another question about user only needs X (keystone) and we install package with clients for all openstack services: Size of keystone client (and any other client I suppose) is ~300Kb I don't think that it's a big difference for the user to install package that is ~300Kb or ~10Mb (unless we are using openstack from Android). It is when most openstack clouds don’t just run keystone. Or nova, or swift. Or when each client acts, smells and behaves differently. It matters a LOT when you’re trying to write applications on top of a mature openstack deployment. From the user perspective I think it's much easier to use client with everything included rather than try to google for client package for some rarely used service. I have actual experience here as a person running a cloud and supporting end-users developers: the overwhelming customer feedback (paid and unpaid (exploratory users)) is that the 22+ clients are confusing, difficult to use, prone to error. There are two ways of resolving this if you’re in my shoes - or in a role where the primary focus is not openstack developers or builders; the first is you coordinate work focusing on developer end user experience upstream, working with openstack and the teams to come up with something that benefits everyone, or, you fork, and build the openstack equivalent to boto / awscli so you can provide a unified “one obvious way to consume openstack clouds”. jesse Regards, Alexei 2014/1/21 Sean Dague s...@dague.net On 01/21/2014 11:54 AM, Renat Akhmerov wrote: On 17 Jan 2014, at 22:00, Jamie Lennox
Re: [openstack-dev] a common client library
On Jan 21, 2014, at 12:59 PM, Alexei Kornienko alexei.kornie...@gmail.commailto:alexei.kornie...@gmail.com wrote: It is when most openstack clouds don’t just run keystone. Or nova, or swift. Or when each client acts, smells and behaves differently. It matters a LOT when you’re trying to write applications on top of a mature openstack deployment. I still don't understand the problem here. Installed packages usually just lie quietly on your disk (which size is now measured in terabytes) and they don't act or stink unless you ask them to. I'm pretty sure that most of the people are not aware of ALL packages installed in their distribution and few packages that are not used won't make it any worse I have actual experience here as a person running a cloud and supporting end-users developers: the overwhelming customer feedback (paid and unpaid (exploratory users)) is that the 22+ clients are confusing, difficult to use, prone to error. There are two ways of resolving this if you’re in my shoes - or in a role where the primary focus is not openstack developers or builders; the first is you coordinate work focusing on developer end user experience upstream, working with openstack and the teams to come up with something that benefits everyone, or, you fork, and build the openstack equivalent to boto / awscli so you can provide a unified “one obvious way to consume openstack clouds”. I agree that many clients can be confusing however I think that actuall problem here is not the clients number but discrepancies in client parameters naming and missing/unclear help. In our work of clients refactoring I will also update bash completition that we have for our clients. So when you use nova start-server you will see options applicable to this command etc. For the unified client we can make a nice commands hierarhcy and I don't think it will be confusing to users for example: $ openstack help nova cloud computing fabric controller glance discovering, registering, retrieving and storing virtual machine images cinder ... 22 records $ openstack help nova OpenStack Nova provides a cloud computing fabric controller, supporting a wide variety of virtualization technologies, including KVM, Xen, LXC, VMware, and more. In addition to its native API, it includes compatibility with the commonly encountered Amazon EC2 and S3 APIs. start-server ... With proper bash completition it should be quite easy to use. Since people are complaining about thread depth: I’ll focus on the blueprint and wiki page for now - we’re disagreeing on basic assumptions and facts here. If you are a developer or a user, trying to build an application that even uses 50% of the currently shipping openstack software; your life is extremely sub optimal. The ideas expressed is to replace the focus of the clients and CLIs from “those who build openstack” to “those that consume openstack” which is a very different profile, knowledge level and user story. 2014/1/21 Jesse Noller jesse.nol...@rackspace.commailto:jesse.nol...@rackspace.com On Jan 21, 2014, at 12:19 PM, Alexei Kornienko alexei.kornie...@gmail.commailto:alexei.kornie...@gmail.com wrote: Hello, I would like to end this requirements talk cause it doesn't make any sense in term of python clients. Initially the discussion was about many clients projects with separate requirements VS single client project with single requirements list. Then I suspect we’re at an impasse; I do plan on proceeding with a new wiki page and blueprint for a unified client, SDK (backend) for end users. I for one do not think things are as clear cut as you make them - and the +1s to the unified client thoughts clearly express the discontent with the current structure packaging. At that moment we should have stop and actually open requirements lists for python clients. Basically all clients have the same requirement (cause they all do the same stuff - sending HTTP requests K.O.) There is absolutely no difference in the situation of many clients vs single client. Answering another question about user only needs X (keystone) and we install package with clients for all openstack services: Size of keystone client (and any other client I suppose) is ~300Kb I don't think that it's a big difference for the user to install package that is ~300Kb or ~10Mb (unless we are using openstack from Android). It is when most openstack clouds don’t just run keystone. Or nova, or swift. Or when each client acts, smells and behaves differently. It matters a LOT when you’re trying to write applications on top of a mature openstack deployment. From the user perspective I think it's much easier to use client with everything included rather than try to google for client package for some rarely used service. I have actual experience here as a person running a cloud and supporting end-users developers: the overwhelming customer feedback (paid and unpaid (exploratory
Re: [openstack-dev] a common client library
On 21 Jan 2014, at 09:07, Jesse Noller jesse.nol...@rackspace.com wrote: Do you use any other platform than Linux? Even donald - one of the python packaging leads and PyPI leads said this is a bad end-user experience for consumers of openstack clouds. That fact that someone (even very smart experience) said something doesn’t mean we should stop thinking forward. I am pretty new to Python though and apologize if I misunderstand things sometimes.___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 21 Jan 2014, at 09:40, Sean Dague s...@dague.net wrote: On 01/21/2014 11:54 AM, Renat Akhmerov wrote: On 17 Jan 2014, at 22:00, Jamie Lennox jamielen...@redhat.com mailto:jamielen...@redhat.com wrote: (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) +1 Renat Akhmerov @ Mirantis Inc. Man, where were you then when we had to spend 3 weeks unwinding global requirements in the gate because pip was figuring it out all kinds of wrong, and we'd do things like uninstall and reinstall python-keystoneclient 6 times during an install. Because after that experience, I'm very anti pip will figure the rest out”. Honestly, I was very far but now I’m much closer :) Because it won't, not in python, where we're talking about libraries that are in the global namespace, where python can only have 1 version of a dependency installed. If the the solution is every openstack project should install a venv for all it's dependencies to get around this issue, then we're talking a different problem (and a different architecture from what we've been trying to do). But I find the idea of having 12 copies of python-keystone client installed on my openstack environment to be distasteful. I see your point. Right now this is really a problem. I hope it’ll be solved someday globally so that python could have more than 1 version of a library. However, some organizational steps could be made to strictly control versions of libraries. Anyway, I admit you’re right here. So come spend a month working on requirements updates in OpenStack gate… Frankly, never had significant experience in that. I would love to :) well you are a braver man than I. :) Nope, it’s hardly true :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
+1 for opening new threads regarding specific questions. On 21 Jan 2014, at 11:54, Renat Akhmerov rakhme...@mirantis.com wrote: On 21 Jan 2014, at 09:40, Sean Dague s...@dague.net wrote: On 01/21/2014 11:54 AM, Renat Akhmerov wrote: On 17 Jan 2014, at 22:00, Jamie Lennox jamielen...@redhat.com mailto:jamielen...@redhat.com wrote: (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) +1 Renat Akhmerov @ Mirantis Inc. Man, where were you then when we had to spend 3 weeks unwinding global requirements in the gate because pip was figuring it out all kinds of wrong, and we'd do things like uninstall and reinstall python-keystoneclient 6 times during an install. Because after that experience, I'm very anti pip will figure the rest out”. Honestly, I was very far but now I’m much closer :) Because it won't, not in python, where we're talking about libraries that are in the global namespace, where python can only have 1 version of a dependency installed. If the the solution is every openstack project should install a venv for all it's dependencies to get around this issue, then we're talking a different problem (and a different architecture from what we've been trying to do). But I find the idea of having 12 copies of python-keystone client installed on my openstack environment to be distasteful. I see your point. Right now this is really a problem. I hope it’ll be solved someday globally so that python could have more than 1 version of a library. However, some organizational steps could be made to strictly control versions of libraries. Anyway, I admit you’re right here. So come spend a month working on requirements updates in OpenStack gate… Frankly, never had significant experience in that. I would love to :) well you are a braver man than I. :) Nope, it’s hardly true :) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 01/19/2014 11:50 PM, Jesse Noller wrote: On Jan 19, 2014, at 5:37 PM, Jamie Lennox jamielen...@redhat.com mailto:jamielen...@redhat.com wrote: On Sat, 2014-01-18 at 09:13 -0500, Doug Hellmann wrote: I like the idea of a fresh start, but I don't think that's incompatible with the other work to clean up the existing clients. That cleanup work could help with creating the backwards compatibility layer, if a new library needs to include one, for example. As far as namespace packages and separate client libraries, I'm torn. It makes sense, and I originally assumed we would want to take that approach. The more I think about it, though, the more I like the approach Dean took with the CLI, creating a single repository with a team responsible for managing consistency in the UI. Doug This *is* the approach Dean took with the CLI. Have a package that provides the CLI but then have the actual work handed off to the individual clients (with quite a lot of glue). And I think many of us are making the argument (or trying to) that the “a lot of glue” approach is wrong and unsustainable for both a unified CLI long term *and especially* for application developers. 100% agree. At some point take a look at the tempest rest client, and you can see how entirely crazy different the APIs are between services - https://github.com/openstack/tempest/blob/master/tempest/common/rest_client.py#L506 (The Tempest client is in no way a paragon of virtues, but by writing our own client we've really discovered how lumpy this API is). So I'm highly supportive of taking all the clients into a single separate program which would produce the official python SDK, as well as a unified CLI for OpenStack. The server programs should just be producing server stacks, that end with the API. I think that would empower a set of people that were most concerned with operator and developer UX to be able to look at OpenStack as a whole. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Sat, 2014-01-18 at 09:13 -0500, Doug Hellmann wrote: I like the idea of a fresh start, but I don't think that's incompatible with the other work to clean up the existing clients. That cleanup work could help with creating the backwards compatibility layer, if a new library needs to include one, for example. As far as namespace packages and separate client libraries, I'm torn. It makes sense, and I originally assumed we would want to take that approach. The more I think about it, though, the more I like the approach Dean took with the CLI, creating a single repository with a team responsible for managing consistency in the UI. Doug This *is* the approach Dean took with the CLI. Have a package that provides the CLI but then have the actual work handed off to the individual clients (with quite a lot of glue). On Sat, Jan 18, 2014 at 1:00 AM, Jamie Lennox jamielen...@redhat.com wrote: I can't see any reason that all of these situations can't be met. We can finally take the openstack pypi namespace, move keystoneclient - openstack.keystone and similar for the other projects. Have them all based upon openstack.base and probably an openstack.transport for transport. For the all-in-one users we can then just have openstack.client which depends on all of the openstack.x projects. This would satisfy the requirement of keeping projects seperate, but having the one entry point for newer users. Similar to the OSC project (which could acutally rely on the new all-in-one). This would also satisfy a lot of the clients who have i know are looking to move to a version 2 and break compatability with some of the crap from the early days. I think what is most important here is deciding what we want from our clients and discussing a common base that we are happy to support - not just renaming the existing ones. (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) Jamie - Original Message - From: Jonathan LaCour jonathan-li...@cleverdevil.org To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Saturday, 18 January, 2014 4:00:58 AM Subject: Re: [openstack-dev] a common client library On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft don...@stufft.io wrote: On Jan 16, 2014, at 4:06 PM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 2:22 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: * Keeping all the clients physically separate/combining them in to a single library. Two things here: * In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) * In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. I agree. Keeping them separate trades user usability for developer usability, I think user usability is a better thing to strive for. 100% agree with this. In order for OpenStack to be its most successful, I believe firmly that a focus on end-users and deployers needs to take the forefront. That means making OpenStack clouds as easy to consume/leverage as possible for users and tool builders, and simplifying/streamlining
Re: [openstack-dev] a common client library
On Jan 19, 2014, at 5:37 PM, Jamie Lennox jamielen...@redhat.commailto:jamielen...@redhat.com wrote: On Sat, 2014-01-18 at 09:13 -0500, Doug Hellmann wrote: I like the idea of a fresh start, but I don't think that's incompatible with the other work to clean up the existing clients. That cleanup work could help with creating the backwards compatibility layer, if a new library needs to include one, for example. As far as namespace packages and separate client libraries, I'm torn. It makes sense, and I originally assumed we would want to take that approach. The more I think about it, though, the more I like the approach Dean took with the CLI, creating a single repository with a team responsible for managing consistency in the UI. Doug This *is* the approach Dean took with the CLI. Have a package that provides the CLI but then have the actual work handed off to the individual clients (with quite a lot of glue). And I think many of us are making the argument (or trying to) that the “a lot of glue” approach is wrong and unsustainable for both a unified CLI long term *and especially* for application developers. On Sat, Jan 18, 2014 at 1:00 AM, Jamie Lennox jamielen...@redhat.commailto:jamielen...@redhat.com wrote: I can't see any reason that all of these situations can't be met. We can finally take the openstack pypi namespace, move keystoneclient - openstack.keystone and similar for the other projects. Have them all based upon openstack.base and probably an openstack.transport for transport. For the all-in-one users we can then just have openstack.client which depends on all of the openstack.x projects. This would satisfy the requirement of keeping projects seperate, but having the one entry point for newer users. Similar to the OSC project (which could acutally rely on the new all-in-one). This would also satisfy a lot of the clients who have i know are looking to move to a version 2 and break compatability with some of the crap from the early days. I think what is most important here is deciding what we want from our clients and discussing a common base that we are happy to support - not just renaming the existing ones. (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) Jamie - Original Message - From: Jonathan LaCour jonathan-li...@cleverdevil.orgmailto:jonathan-li...@cleverdevil.org To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Sent: Saturday, 18 January, 2014 4:00:58 AM Subject: Re: [openstack-dev] a common client library On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft don...@stufft.iomailto:don...@stufft.io wrote: On Jan 16, 2014, at 4:06 PM, Jesse Noller jesse.nol...@rackspace.commailto:jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 2:22 PM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: * Keeping all the clients physically separate/combining them in to a single library. Two things here: * In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) * In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. I agree. Keeping them separate trades user usability for developer usability, I think user usability is a better thing to strive for. 100% agree with this. In order for OpenStack to be its most successful, I believe firmly that a focus on end-users and deployers needs to take the forefront. That means making OpenStack clouds as easy to consume/leverage as possible for users and tool builders, and simplifying/streamlining as much as possible. I think that a single, common client project, based upon package namespaces, with a unified, cohesive feel is a big step in this direction. -- Jonathan
Re: [openstack-dev] a common client library
On Fri, Jan 17, 2014 at 11:03 PM, John Utz john@wdc.com wrote: Outlook Web MUA, forgive the top post. :-( While a single import line that brings all the good stuff in at one shot is very convenient for the creation of an application, it would muddy the security picture *substantially* for the exact type of developer\customer that you would be targeting with this sort of syntactic sugar. As Jesse alludes to below, the expanding tree of dependencies would be masked by the aggregation. So, most likely, they would be pulling in vast numbers of things that they don't require to get their simple app done (there's an idea! an eclipse plugin that helpfully points out all the things that you are *not* using and offers to redo your imports for you :-) ). I'm not sure what vast numbers of things you mean. The point is to make one thing to talk to an OpenStack cloud consistently, instead of a separate library for every facet of the cloud. Surely building on a common code base will have the opposite effect you suggest -- it should reduce the number of dependencies, and make it easier to track security updates in those dependencies. Doug As a result, when a security defect is published concerning one of those hidden dependencies, they will not have any reason to think that it effects them. just my us$0.02; johnu From: Jesse Noller [jesse.nol...@rackspace.com] Sent: Thursday, January 16, 2014 5:42 PM To: OpenStack Development Mailing List (not for usage questions) Cc: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] a common client library On Jan 16, 2014, at 4:59 PM, Renat Akhmerov rakhme...@mirantis.com mailto:rakhme...@mirantis.com wrote: On 16 Jan 2014, at 13:06, Jesse Noller jesse.nol...@rackspace.commailto: jesse.nol...@rackspace.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: * Keeping all the clients physically separate/combining them in to a single library. Two things here: * In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) * In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. You may be right but not sure that adding another line into requirements.txt is a huge loss of usability. It is when that 1 dependency pulls in 6 others that pull in 10 more - every little barrier or potential failure from the inability to make a static binary to how each tool acts different is a paper cut of frustration to an end user. Most of the time the clients don't even properly install because of dependencies on setuptools plugins and other things. For developers (as I've said) the story is worse: you have potentially 22+ individual packages and their dependencies to deal with if they want to use a complete openstack install from their code. So it doesn't boil down to just 1 dependency: it's a long laundry list of things that make consumers' lives more difficult and painful. This doesn't even touch on the fact there aren't blessed SDKs or tools pointing users to consume openstack in their preferred programming language. Shipping an API isn't enough - but it can be fixed easily enough. Renat Akhmerov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
I like the idea of a fresh start, but I don't think that's incompatible with the other work to clean up the existing clients. That cleanup work could help with creating the backwards compatibility layer, if a new library needs to include one, for example. As far as namespace packages and separate client libraries, I'm torn. It makes sense, and I originally assumed we would want to take that approach. The more I think about it, though, the more I like the approach Dean took with the CLI, creating a single repository with a team responsible for managing consistency in the UI. Doug On Sat, Jan 18, 2014 at 1:00 AM, Jamie Lennox jamielen...@redhat.comwrote: I can't see any reason that all of these situations can't be met. We can finally take the openstack pypi namespace, move keystoneclient - openstack.keystone and similar for the other projects. Have them all based upon openstack.base and probably an openstack.transport for transport. For the all-in-one users we can then just have openstack.client which depends on all of the openstack.x projects. This would satisfy the requirement of keeping projects seperate, but having the one entry point for newer users. Similar to the OSC project (which could acutally rely on the new all-in-one). This would also satisfy a lot of the clients who have i know are looking to move to a version 2 and break compatability with some of the crap from the early days. I think what is most important here is deciding what we want from our clients and discussing a common base that we are happy to support - not just renaming the existing ones. (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) Jamie - Original Message - From: Jonathan LaCour jonathan-li...@cleverdevil.org To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Saturday, 18 January, 2014 4:00:58 AM Subject: Re: [openstack-dev] a common client library On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft don...@stufft.io wrote: On Jan 16, 2014, at 4:06 PM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 2:22 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: * Keeping all the clients physically separate/combining them in to a single library. Two things here: * In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) * In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. I agree. Keeping them separate trades user usability for developer usability, I think user usability is a better thing to strive for. 100% agree with this. In order for OpenStack to be its most successful, I believe firmly that a focus on end-users and deployers needs to take the forefront. That means making OpenStack clouds as easy to consume/leverage as possible for users and tool builders, and simplifying/streamlining as much as possible. I think that a single, common client project, based upon package namespaces, with a unified, cohesive feel is a big step in this direction. -- Jonathan LaCour ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 18, 2014, at 8:13 AM, Doug Hellmann doug.hellm...@dreamhost.commailto:doug.hellm...@dreamhost.com wrote: I like the idea of a fresh start, but I don't think that's incompatible with the other work to clean up the existing clients. That cleanup work could help with creating the backwards compatibility layer, if a new library needs to include one, for example. As far as namespace packages and separate client libraries, I'm torn. It makes sense, and I originally assumed we would want to take that approach. The more I think about it, though, the more I like the approach Dean took with the CLI, creating a single repository with a team responsible for managing consistency in the UI. Doug I’m going to pursue the latter - but at the same time the current effort to clean things up seems to be running afoul of the keystone client changes in flight? On Sat, Jan 18, 2014 at 1:00 AM, Jamie Lennox jamielen...@redhat.commailto:jamielen...@redhat.com wrote: I can't see any reason that all of these situations can't be met. We can finally take the openstack pypi namespace, move keystoneclient - openstack.keystone and similar for the other projects. Have them all based upon openstack.base and probably an openstack.transport for transport. For the all-in-one users we can then just have openstack.client which depends on all of the openstack.x projects. This would satisfy the requirement of keeping projects seperate, but having the one entry point for newer users. Similar to the OSC project (which could acutally rely on the new all-in-one). This would also satisfy a lot of the clients who have i know are looking to move to a version 2 and break compatability with some of the crap from the early days. I think what is most important here is deciding what we want from our clients and discussing a common base that we are happy to support - not just renaming the existing ones. (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) Jamie - Original Message - From: Jonathan LaCour jonathan-li...@cleverdevil.orgmailto:jonathan-li...@cleverdevil.org To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Sent: Saturday, 18 January, 2014 4:00:58 AM Subject: Re: [openstack-dev] a common client library On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft don...@stufft.iomailto:don...@stufft.io wrote: On Jan 16, 2014, at 4:06 PM, Jesse Noller jesse.nol...@rackspace.commailto:jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 2:22 PM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: * Keeping all the clients physically separate/combining them in to a single library. Two things here: * In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) * In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. I agree. Keeping them separate trades user usability for developer usability, I think user usability is a better thing to strive for. 100% agree with this. In order for OpenStack to be its most successful, I believe firmly that a focus on end-users and deployers needs to take the forefront. That means making OpenStack clouds as easy to consume/leverage as possible for users and tool builders, and simplifying/streamlining as much as possible. I think that a single, common client project, based upon package namespaces, with a unified, cohesive feel is a big step in this direction. -- Jonathan LaCour ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list
Re: [openstack-dev] a common client library
On Jan 18, 2014, at 12:00 AM, Jamie Lennox jamielen...@redhat.com wrote: I can't see any reason that all of these situations can't be met. We can finally take the openstack pypi namespace, move keystoneclient - openstack.keystone and similar for the other projects. Have them all based upon openstack.base and probably an openstack.transport for transport. For the all-in-one users we can then just have openstack.client which depends on all of the openstack.x projects. This would satisfy the requirement of keeping projects seperate, but having the one entry point for newer users. Similar to the OSC project (which could acutally rely on the new all-in-one). This would also satisfy a lot of the clients who have i know are looking to move to a version 2 and break compatability with some of the crap from the early days. I think what is most important here is deciding what we want from our clients and discussing a common base that we are happy to support - not just renaming the existing ones. (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) You’re assuming: 1: Pip works when installing the entire dependency graph (it often doesn’t) 2: For some of these requirements, the user has a compiler installed (they don’t) 3: Installing 1 “meta package” that install N+K dependencies makes end user consumers happy (it doesn’t) 4: All of these dependencies make shipping a single binary deployment easier (it doesn’t) 5: Installing and using all of these things makes using openstack within my code conceptually simpler (it doesn’t) We can start with *not* renaming the sub clients (meaning) collapsing them into the singular namespace; but the problem is that every one of those sub dependencies is potential liability to someone using this single client. If yes, we could only target fedora, and rely on yum rpm, I’d agree with you - but for python application dependencies across multiple OSes and developers doing ci/cd using these systems I can’t. I also don’t want user to stumble into the nuanced vagaries of the sub-clients when writing application code; writing glue code to bind them all together does work very well (we know this from experience). Jamie - Original Message - From: Jonathan LaCour jonathan-li...@cleverdevil.org To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Saturday, 18 January, 2014 4:00:58 AM Subject: Re: [openstack-dev] a common client library On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft don...@stufft.io wrote: On Jan 16, 2014, at 4:06 PM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 2:22 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: * Keeping all the clients physically separate/combining them in to a single library. Two things here: * In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) * In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. I agree. Keeping them separate trades user usability for developer usability, I think user usability is a better thing to strive for. 100% agree with this. In order for OpenStack to be its most successful, I believe firmly that a focus on end-users and deployers needs to take the forefront. That means making OpenStack clouds as easy to consume/leverage as possible for users and tool builders, and simplifying/streamlining as much as possible. I think that a single, common client project, based upon package namespaces, with a unified, cohesive feel is a big step in this direction. -- Jonathan LaCour ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev
Re: [openstack-dev] a common client library
On 01/18/2014 01:06 AM, Robert Collins wrote: On 17 January 2014 09:22, Renat Akhmerov rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Has anyone ever considered an idea of generating a fully functional REST client automatically based on an API specification (WADL could be used for that)? Not sure how convenient it would be, it really depends on a particular implementation, but as an idea it could be at least thought of. Sounds a little bit crazy though, I recognize it :). Launchpadlib which builds on wadllib did *exactly* that. It worked fairly well with the one caveat that it fell into the ORM trap - just in time lookups for everything with crippling roundtrips. -1 if the answer to anything is WADL. It's terrible, and an abandoned sun rfc at this point. We've got some real progress in getting JSON schema in place in Nova (for validation, but it's incremental steps, and good ones), which I think is much more fruitful. I also think auto generated clients have a lot of challenges in the same way that full javascript pages in browsers have. If you screw up in a subtle way you can just completely disable your clients from connecting to your server entirely (or block them from using bits of the lesser used function path because a minor bit of schema fat fingered). So without a ton of additional verification on that path, I wouldn't want that anywhere near openstack. Especially with vendor extensions, which mean that enabling a network driver might break all your clients. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 18, 2014, at 12:58 AM, Robert Collins robe...@robertcollins.net wrote: Out of interest - whats the overhead of running tls compression against compressed data? Is it really noticable? The overhead doesn’t really matter much as you want TLS Compression disabled because of CRIME anyways. Most Linux distros and such ship with it disabled by default now IIRC. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 18, 2014, at 9:58 AM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 18, 2014, at 12:00 AM, Jamie Lennox jamielen...@redhat.com wrote: I can't see any reason that all of these situations can't be met. We can finally take the openstack pypi namespace, move keystoneclient - openstack.keystone and similar for the other projects. Have them all based upon openstack.base and probably an openstack.transport for transport. For the all-in-one users we can then just have openstack.client which depends on all of the openstack.x projects. This would satisfy the requirement of keeping projects seperate, but having the one entry point for newer users. Similar to the OSC project (which could acutally rely on the new all-in-one). This would also satisfy a lot of the clients who have i know are looking to move to a version 2 and break compatability with some of the crap from the early days. I think what is most important here is deciding what we want from our clients and discussing a common base that we are happy to support - not just renaming the existing ones. (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) You’re assuming: 1: Pip works when installing the entire dependency graph (it often doesn’t) 2: For some of these requirements, the user has a compiler installed (they don’t) 3: Installing 1 “meta package” that install N+K dependencies makes end user consumers happy (it doesn’t) 4: All of these dependencies make shipping a single binary deployment easier (it doesn’t) 5: Installing and using all of these things makes using openstack within my code conceptually simpler (it doesn’t) We can start with *not* renaming the sub clients (meaning) collapsing them into the singular namespace; but the problem is that every one of those sub dependencies is potential liability to someone using this single client. If yes, we could only target fedora, and rely on yum rpm, I’d agree with you - but for python application dependencies across multiple OSes and developers doing ci/cd using these systems I can’t. I also don’t want user to stumble into the nuanced vagaries of the sub-clients when writing application code; writing glue code to bind them all together does work very well (we know this from experience). As much as I would like to say (with my pip developer and PyPI Admin hat on) that depending on 22+ libraries in a single client will be a seamless experience for end users I can’t in good faith say that it would be yet. We’re working on trying to make that true but honestly each dependency in a graph does introduce risk. As of right now there is no real dependency solver in pip, so if someone depends on the openstack client themselves, and then depends on something else that depends on one of the sub clients as well if those version specs don’t match up there is a very good chance that the end user will run into a very confusing message at runtime. Openstack itself has run into this problem and it was a big motivator for the global requirements project. Additionally it’s not uncommon for users to have policy driven requirements that require them to get every dependency they pull in checked for compliance (License, security etc). Having a 22+ node dependency graph makes this issue much harder in general. I also believe in general it’s asking for user confusion. It’s much simpler to document a single way of doing it, however splitting the clients up and then wrapping them with a single “openstack” client means that you have at least two ways of doing something; The direct “use just a single library” approach and the “use the openstack wrapper” approach. Don’t underestimate the confusion this will cause end users. Keeping them all under one project will make it far easier to have a cohesive API amongst all the various services, it will reduce duplication of efforts, as well as make it easier to track security updates and I believe a wholly superior end user experience. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 19 January 2014 04:48, Sean Dague s...@dague.net wrote: On 01/18/2014 01:06 AM, Robert Collins wrote: Launchpadlib which builds on wadllib did *exactly* that. It worked fairly well with the one caveat that it fell into the ORM trap - just in time lookups for everything with crippling roundtrips. -1 if the answer to anything is WADL. It's terrible, and an abandoned sun rfc at this point. We've got some real progress in getting JSON schema in place in Nova (for validation, but it's incremental steps, and good ones), which I think is much more fruitful. I also think auto generated clients have a lot of challenges in the same way that full javascript pages in browsers have. If you screw up in a subtle way you can just completely disable your clients from connecting to your server entirely (or block them from using bits of the lesser used function path because a minor bit of schema fat fingered). So without a ton of additional verification on that path, I wouldn't want that anywhere near openstack. Especially with vendor extensions, which mean that enabling a network driver might break all your clients. To be clear: I'm not advocating this approach. Just answering the question 'has it been tried', with 'yes, and this is what was wrong'. The actual code generation and execution worked very well, and most features added to the server side APIs were available immediately in the client with no upgrades. However any new /type/ of feature required changes to the compiler. Personally, I think hand crafted clients have a much better feel to them - idiomatic to the language they are in, easier to predict, and much easier to understand. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. I agree. Keeping them separate trades user usability for developer usability, I think user usability is a better thing to strive for. I don't understand how multiple independent code bases with a lot of overlapping code/logic is a win for developers. The more we can move to single shared code the easier code comprehension and maintenance becomes. From a software engineering perspective the amount of duplicated code/logic in OpenStack is worrisome. Iterating towards common code seems like a huge developer win as well as greatly enhancing robustness in the process. -- John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
It seems we have two target audiences here. Developers who work on OpenStack and developers who write apps to use it. In the long run I think we need to optimize for both of these groups. If we want developers to write applications to use OpenStack in python we likely need a common python SDK. Note, I'm not a fan of the term client because it's not the common language for this group of developers. On Fri, Jan 17, 2014 at 10:26 AM, John Dennis jden...@redhat.com wrote: Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. I agree. Keeping them separate trades user usability for developer usability, I think user usability is a better thing to strive for. I don't understand how multiple independent code bases with a lot of overlapping code/logic is a win for developers. The more we can move to single shared code the easier code comprehension and maintenance becomes. From a software engineering perspective the amount of duplicated code/logic in OpenStack is worrisome. Iterating towards common code seems like a huge developer win as well as greatly enhancing robustness in the process. -- John ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft don...@stufft.io wrote: On Jan 16, 2014, at 4:06 PM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 2:22 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: - Keeping all the clients physically separate/combining them in to a single library. Two things here: - In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) - In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. I agree. Keeping them separate trades user usability for developer usability, I think user usability is a better thing to strive for. 100% agree with this. In order for OpenStack to be its most successful, I believe firmly that a focus on end-users and deployers needs to take the forefront. That means making OpenStack clouds as easy to consume/leverage as possible for users and tool builders, and simplifying/streamlining as much as possible. I think that a single, common client project, based upon package namespaces, with a unified, cohesive feel is a big step in this direction. -- Jonathan LaCour ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 5:42 PM, Jesse Noller jesse.nol...@rackspace.comwrote: On Jan 16, 2014, at 4:59 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 16 Jan 2014, at 13:06, Jesse Noller jesse.nol...@rackspace.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: - Keeping all the clients physically separate/combining them in to a single library. Two things here: - In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) - In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. You may be right but not sure that adding another line into requirements.txt is a huge loss of usability. It is when that 1 dependency pulls in 6 others that pull in 10 more - every little barrier or potential failure from the inability to make a static binary to how each tool acts different is a paper cut of frustration to an end user. Most of the time the clients don't even properly install because of dependencies on setuptools plugins and other things. For developers (as I've said) the story is worse: you have potentially 22+ individual packages and their dependencies to deal with if they want to use a complete openstack install from their code. So it doesn't boil down to just 1 dependency: it's a long laundry list of things that make consumers' lives more difficult and painful. This doesn't even touch on the fact there aren't blessed SDKs or tools pointing users to consume openstack in their preferred programming language. Shipping an API isn't enough - but it can be fixed easily enough. +100 OpenStack must be attractive to our end users (developers and deployers), as I mentioned earlier. Let's make it as simple as pip install openstack if at all possible! -- Jonathan LaCour ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 17 Jan 2014, at 10:04, Jonathan LaCour jonathan-li...@cleverdevil.org wrote: pip install openstack That would be awesome :) Renat Akhmerov @ Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
Outlook Web MUA, forgive the top post. :-( While a single import line that brings all the good stuff in at one shot is very convenient for the creation of an application, it would muddy the security picture *substantially* for the exact type of developer\customer that you would be targeting with this sort of syntactic sugar. As Jesse alludes to below, the expanding tree of dependencies would be masked by the aggregation. So, most likely, they would be pulling in vast numbers of things that they don't require to get their simple app done (there's an idea! an eclipse plugin that helpfully points out all the things that you are *not* using and offers to redo your imports for you :-) ). As a result, when a security defect is published concerning one of those hidden dependencies, they will not have any reason to think that it effects them. just my us$0.02; johnu From: Jesse Noller [jesse.nol...@rackspace.com] Sent: Thursday, January 16, 2014 5:42 PM To: OpenStack Development Mailing List (not for usage questions) Cc: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] a common client library On Jan 16, 2014, at 4:59 PM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: On 16 Jan 2014, at 13:06, Jesse Noller jesse.nol...@rackspace.commailto:jesse.nol...@rackspace.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: * Keeping all the clients physically separate/combining them in to a single library. Two things here: * In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) * In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. You may be right but not sure that adding another line into requirements.txt is a huge loss of usability. It is when that 1 dependency pulls in 6 others that pull in 10 more - every little barrier or potential failure from the inability to make a static binary to how each tool acts different is a paper cut of frustration to an end user. Most of the time the clients don't even properly install because of dependencies on setuptools plugins and other things. For developers (as I've said) the story is worse: you have potentially 22+ individual packages and their dependencies to deal with if they want to use a complete openstack install from their code. So it doesn't boil down to just 1 dependency: it's a long laundry list of things that make consumers' lives more difficult and painful. This doesn't even touch on the fact there aren't blessed SDKs or tools pointing users to consume openstack in their preferred programming language. Shipping an API isn't enough - but it can be fixed easily enough. Renat Akhmerov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 17 January 2014 06:39, Mark Washenberger mark.washenber...@markwash.net wrote: There's a few more items here that are needed for glance to be able to work with requests (which we really really want). 1) Support for 100-expect-continue is probably going to be required in glance as well as swift Is this currently supported? If not, frankly, I wouldn't bother. The semantics in HTTP/2 are much better - I'd aim straight at that. 2) Support for turning off tls/ssl compression (our streams are already compressed) Out of interest - whats the overhead of running tls compression against compressed data? Is it really noticable? -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
I can't see any reason that all of these situations can't be met. We can finally take the openstack pypi namespace, move keystoneclient - openstack.keystone and similar for the other projects. Have them all based upon openstack.base and probably an openstack.transport for transport. For the all-in-one users we can then just have openstack.client which depends on all of the openstack.x projects. This would satisfy the requirement of keeping projects seperate, but having the one entry point for newer users. Similar to the OSC project (which could acutally rely on the new all-in-one). This would also satisfy a lot of the clients who have i know are looking to move to a version 2 and break compatability with some of the crap from the early days. I think what is most important here is deciding what we want from our clients and discussing a common base that we are happy to support - not just renaming the existing ones. (I don't buy the problem with large amounts of dependencies, if you have a meta-package you just have one line in requirements and pip will figure the rest out.) Jamie - Original Message - From: Jonathan LaCour jonathan-li...@cleverdevil.org To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Saturday, 18 January, 2014 4:00:58 AM Subject: Re: [openstack-dev] a common client library On Thu, Jan 16, 2014 at 1:23 PM, Donald Stufft don...@stufft.io wrote: On Jan 16, 2014, at 4:06 PM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 2:22 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: * Keeping all the clients physically separate/combining them in to a single library. Two things here: * In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) * In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. I agree. Keeping them separate trades user usability for developer usability, I think user usability is a better thing to strive for. 100% agree with this. In order for OpenStack to be its most successful, I believe firmly that a focus on end-users and deployers needs to take the forefront. That means making OpenStack clouds as easy to consume/leverage as possible for users and tool builders, and simplifying/streamlining as much as possible. I think that a single, common client project, based upon package namespaces, with a unified, cohesive feel is a big step in this direction. -- Jonathan LaCour ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 17 January 2014 06:57, Mark Washenberger
mark.washenber...@markwash.net wrote:
Just throwing this out there because it seems relevant to client design.
As we've been looking at porting clients to using v2 of the Images API, its
seems more and more to me that including the *server* version in the main
import path is a real obstacle.
IMO any future client libs should write library interfaces based on the
peculiarities of user needs, not based on the vagaries of the server
version. So as a user of this library I would do something like:
1 from openstack.api import images
2 client = images.make_me_a_client(auth_url, etcetera) # all version
negotiation is happening here
3 client.list_images() # works more or less same no matter who I'm
talking to
Now, there would still likely be hidden implementation code that is
different per server version and which is instantiated in line 2 above, and
maybe that's the library path stuff you are talking about.
That design seems guaranteed to behave somewhat poorly (e.g. fail to
upgrade) when servers are upgraded - for short lived processes like
'nova boot' that doesn't matter, but for software running in a daemon
- e.g. in nova-api talking to neutron - that seems much more likely to
be a problem.
I think the pseudo code is fine, but client shouldn't be a concrete
version locked client, rather a proxy object that can revalidate the
version every {sensible time period} and/or observe HTTP headers to
detect when upgrades are possible (or downgrades are required).
-Rob
--
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 17 January 2014 08:03, Alexei Kornienko alexei.kornie...@gmail.com wrote: Hello Joe, 2)Another option would be to follow waterfall process and create a solid library interface before including it to all client projects. However such approach this period can take unknown amount of time and can be easily failed during integration stage cause requirements change or some other reason. Please let me know what you think. Given how fast OpenStack moves, I think waterfall is a non-starter. I'd do a library from the start, and just do backwards compat - it's really not that hard :) -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 17 January 2014 09:22, Renat Akhmerov rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Has anyone ever considered an idea of generating a fully functional REST client automatically based on an API specification (WADL could be used for that)? Not sure how convenient it would be, it really depends on a particular implementation, but as an idea it could be at least thought of. Sounds a little bit crazy though, I recognize it :). Launchpadlib which builds on wadllib did *exactly* that. It worked fairly well with the one caveat that it fell into the ORM trap - just in time lookups for everything with crippling roundtrips. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 15/01/14 21:35 +, Jesse Noller wrote: On Jan 15, 2014, at 1:37 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 *raises hand* Me me! I’ve been talking to many contributors about the Developer Experience stuff I emailed out prior to the holidays and I was starting blueprint work, but this is a great pointer. I’m going to have to sync up with Alexei. I think solving this for openstack developers and maintainers as the blueprint says is a big win in terms of code reuse / maintenance and consistent but more so for *end-user developers* consuming openstack clouds. Some background - there’s some terminology mismatch but the rough idea is the same: * A centralized “SDK” (Software Development Kit) would be built condensing the common code and logic and operations into a single namespace. * This SDK would be able to be used by “downstream” CLIs - essentially the CLIs become a specialized front end - and in some cases, only an argparse or cliff front-end to the SDK methods located in the (for example) openstack.client.api.compute * The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived clients - all of the openstack.client.api.** classes would accept an Auth object to delegate management / mocking of the Auth / service catalog stuff to. This means developers building applications (say for example, horizon) don’t need to worry about token/expired authentication/etc. * Simplify the dependency graph code for the existing tools to enable single binary installs (py2exe, py2app, etc) for end users of the command line tools. Short version: if a developer wants to consume an openstack cloud; the would have a single SDK with minimal dependencies and import from a single namespace. An example application might look like: from openstack.api import AuthV2 from openstack.api import ComputeV2 myauth = AuthV2(…., connect=True) compute = ComputeV2(myauth) compute.list_flavors() I know this is an example but, could we leave the version out of the class name? Having something like: from openstack.api.v2 import Compute or from openstack.compute.v2 import Instance (just made that up) for marconi we're using the later. This greatly improves the developer experience both internal to openstack and externally. Currently OpenStack has 22+ (counting stackforge) potential libraries a developer may need to install to use a full deployment of OpenStack: * python-keystoneclient (identity) * python-glanceclient (image) * python-novaclient (compute) * python-troveclient (database) * python-neutronclient (network) * python-ironicclient (bare metal) * python-heatclient (orchestration) * python-cinderclient (block storage) * python-ceilometerclient (telemetry, metrics billing) * python-swiftclient (object storage) * python-savannaclient (big data) * python-openstackclient (meta client package) * python-marconiclient (queueing) * python-tuskarclient (tripleo / management) * python-melangeclient (dead) * python-barbicanclient (secrets) * python-solumclient (ALM) * python-muranoclient (application catalog) * python-manilaclient (shared filesystems) * python-libraclient (load balancers) * python-climateclient (reservations) * python-designateclient (Moniker/DNS) If you exclude the above and look on PyPI: On PyPi (client libraries/SDKs only, excluding the above - not maintained by openstack): * hpcloud-auth-openstack 1.0 * python-openstackclient 0.2.2 * rackspace-auth-openstack 1.1 * posthaste 0.2.2 * pyrax 1.6.2 * serverherald 0.0.1 * warm 0.3.1 * vaporize 0.3.2 * swiftsc (https://github.com/mkouhei/swiftsc) * bookofnova 0.007 * nova-adminclient 0.1.8 * python-quantumclient 2.2.4.3 * python-stackhelper 0.0.7.1.gcab1eb0 * swift-bench 1.0 * swiftly 1.12 * txAWS 0.2.3 * cfupload 0.5.1 * python-reddwarfclient 0.1.2 * python-automationclient 1.2.1 * rackspace-glanceclient 0.9 * rackspace-novaclient 1.4 If you ignore PyPI and just want to install the base say - 7 services, each one of the python-** clients has its own dependency tree and may or may not build from one of the others. If a vendor wants to extend any of them, it’s basically a fork instead of a clean plugin system. On the CLI front - this would *greatly* simplify the work openstackclient needs
Re: [openstack-dev] a common client library
Hi Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? (I'm thinking reduced load for packagers, simpler installation for users, etc) Cheers, -- Chris Jones On 15 Jan 2014, at 19:37, Doug Hellmann doug.hellm...@dreamhost.com wrote: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 2:09 AM, Flavio Percoco fla...@redhat.com wrote: On 15/01/14 21:35 +, Jesse Noller wrote: On Jan 15, 2014, at 1:37 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 *raises hand* Me me! I’ve been talking to many contributors about the Developer Experience stuff I emailed out prior to the holidays and I was starting blueprint work, but this is a great pointer. I’m going to have to sync up with Alexei. I think solving this for openstack developers and maintainers as the blueprint says is a big win in terms of code reuse / maintenance and consistent but more so for *end-user developers* consuming openstack clouds. Some background - there’s some terminology mismatch but the rough idea is the same: * A centralized “SDK” (Software Development Kit) would be built condensing the common code and logic and operations into a single namespace. * This SDK would be able to be used by “downstream” CLIs - essentially the CLIs become a specialized front end - and in some cases, only an argparse or cliff front-end to the SDK methods located in the (for example) openstack.client.api.compute * The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived clients - all of the openstack.client.api.** classes would accept an Auth object to delegate management / mocking of the Auth / service catalog stuff to. This means developers building applications (say for example, horizon) don’t need to worry about token/expired authentication/etc. * Simplify the dependency graph code for the existing tools to enable single binary installs (py2exe, py2app, etc) for end users of the command line tools. Short version: if a developer wants to consume an openstack cloud; the would have a single SDK with minimal dependencies and import from a single namespace. An example application might look like: from openstack.api import AuthV2 from openstack.api import ComputeV2 myauth = AuthV2(…., connect=True) compute = ComputeV2(myauth) compute.list_flavors() I know this is an example but, could we leave the version out of the class name? Having something like: from openstack.api.v2 import Compute or from openstack.compute.v2 import Instance (just made that up) for marconi we're using the later. Definitely; it should be based on namespaces. This greatly improves the developer experience both internal to openstack and externally. Currently OpenStack has 22+ (counting stackforge) potential libraries a developer may need to install to use a full deployment of OpenStack: * python-keystoneclient (identity) * python-glanceclient (image) * python-novaclient (compute) * python-troveclient (database) * python-neutronclient (network) * python-ironicclient (bare metal) * python-heatclient (orchestration) * python-cinderclient (block storage) * python-ceilometerclient (telemetry, metrics billing) * python-swiftclient (object storage) * python-savannaclient (big data) * python-openstackclient (meta client package) * python-marconiclient (queueing) * python-tuskarclient (tripleo / management) * python-melangeclient (dead) * python-barbicanclient (secrets) * python-solumclient (ALM) * python-muranoclient (application catalog) * python-manilaclient (shared filesystems) * python-libraclient (load balancers) * python-climateclient (reservations) * python-designateclient (Moniker/DNS) If you exclude the above and look on PyPI: On PyPi (client libraries/SDKs only, excluding the above - not maintained by openstack): * hpcloud-auth-openstack 1.0 * python-openstackclient 0.2.2 * rackspace-auth-openstack 1.1 * posthaste 0.2.2 * pyrax 1.6.2 * serverherald 0.0.1 * warm 0.3.1 * vaporize 0.3.2 * swiftsc (https://github.com/mkouhei/swiftsc) * bookofnova 0.007 * nova-adminclient 0.1.8 * python-quantumclient 2.2.4.3 * python-stackhelper 0.0.7.1.gcab1eb0 * swift-bench 1.0 * swiftly 1.12 * txAWS 0.2.3 * cfupload 0.5.1 * python-reddwarfclient 0.1.2 * python-automationclient 1.2.1 * rackspace-glanceclient 0.9 * rackspace-novaclient 1.4 If you ignore PyPI and just want to install the base say - 7 services, each one of the python-** clients has its own dependency
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 5:42 AM, Chris Jones c...@tenshu.netmailto:c...@tenshu.net wrote: Hi Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? (I'm thinking reduced load for packagers, simpler installation for users, etc) Cheers, -- Chris Jones Based on the email I sent; ideally we'd expand the blueprint to include these details and speak to the PTLs for each project. Ideally (IMO) yes - long term they would be collapsed into one or two packages to install. However at very least the code can be made so that the core functionality is in the common/single sdk backend and openstackcli, and each project CLI can derive their named/branded project specific one for that. For standalone installs of nova or swift for example; not pushing users to use the entire openstackclient makes sense, instead guiding users to use a tool dedicated to that. There are pros and cons on both sides, meeting in the middle at least collapses the heavy lifting into the common code base. On 15 Jan 2014, at 19:37, Doug Hellmann doug.hellm...@dreamhost.commailto:doug.hellm...@dreamhost.com wrote: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones c...@tenshu.net wrote: Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? that's what openstackclient is here for https://github.com/openstack/python-openstackclient ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah chmo...@enovance.commailto:chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones c...@tenshu.netmailto:c...@tenshu.net wrote: Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? that's what openstackclient is here for https://github.com/openstack/python-openstackclient After speaking with people working on OSC and looking at the code base in depth; I don’t think this addresses what Chris is implying: OSC wraps the individual CLIs built by each project today, instead of the inverse: a common backend that the individual CLIs can wrap - the latter is an important distinction as currently, building a single binary install of OSC for say, Windows is difficult given the dependency tree incurred by each of the wrapped CLIs, difference in dependencies, structure, etc. Also, wrapping a series of inconsistent back end Client classes / functions / methods means that the layer that presents a consistent user interface (OSC) to the user is made more complex juggling names/renames/commands/etc. In the inverted case of what we have today (single backend); as a developer of user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to: from openstack.common.api import Auth from openstack.common.api import Compute from openstack.common.util import cli_tools my_cli = cli_tools.build(…) def my_command(cli): compute = Compute(Auth(cli.tentant…, connect=True)) compute.list_flavors() This would mean that *even if the individual clients needed or wanted to keep their specific CLIs, they would be able to use a not “least common denominator” back end (each service can have a rich common.api.compute.py or api.compute/client.py and extend where needed. However tools like horizon / openstackclient can choose not to leverage the “power user/operator/admin” components and present a simplified user interface. I’m working on a wiki page + blueprint to brainstorm how we could accomplish this based off of what work is in flight today (see doug’s linked blueprint) and sussing out a layout / API strawman for discussion. Some of the additions that came out of this email threads and others: 1. Common backend should provide / offer caching utilities 2. Auth retries need to be handled by the auth object, and each sub-project delegates to the auth object to manage that. 3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 9:07 AM, Joe Gordon joe.gord...@gmail.commailto:joe.gord...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller jesse.nol...@rackspace.commailto:jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah chmo...@enovance.commailto:chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones c...@tenshu.netmailto:c...@tenshu.net wrote: Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? that's what openstackclient is here for https://github.com/openstack/python-openstackclient After speaking with people working on OSC and looking at the code base in depth; I don’t think this addresses what Chris is implying: OSC wraps the individual CLIs built by each project today, instead of the inverse: a common backend that the individual CLIs can wrap - the latter is an important distinction as currently, building a single binary install of OSC for say, Windows is difficult given the dependency tree incurred by each of the wrapped CLIs, difference in dependencies, structure, etc. Also, wrapping a series of inconsistent back end Client classes / functions / methods means that the layer that presents a consistent user interface (OSC) to the user is made more complex juggling names/renames/commands/etc. In the inverted case of what we have today (single backend); as a developer of user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to: from openstack.common.api import Auth from openstack.common.api import Compute from openstack.common.util import cli_tools my_cli = cli_tools.build(…) def my_command(cli): compute = Compute(Auth(cli.tentant…, connect=True)) compute.list_flavors() This would mean that *even if the individual clients needed or wanted to keep their specific CLIs, they would be able to use a not “least common denominator” back end (each service can have a rich common.api.compute.pyhttp://common.api.compute.py/ or api.compute/client.py and extend where needed. However tools like horizon / openstackclient can choose not to leverage the “power user/operator/admin” components and present a simplified user interface. I’m working on a wiki page + blueprint to brainstorm how we could accomplish this based off of what work is in flight today (see doug’s linked blueprint) and sussing out a layout / API strawman for discussion. Some of the additions that came out of this email threads and others: 1. Common backend should provide / offer caching utilities 2. Auth retries need to be handled by the auth object, and each sub-project delegates to the auth object to manage that. 3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing I am happy to see this work being done, there is definitely a lot of work to be done on the clients. This blueprint sounds like its still being fleshed out, so I am wondering what the value is of the current patches https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z Those patches mainly sync cliutils and apiutils from oslo into the assorted clients. But if this blueprint is about the python API and not the CLI (as that would be the openstack-pythonclient), why sync in apiutils? Also does this need to go through oslo-incubator or can this start out as a library? Making this a library earlier on will reduce the number of patches needed to get 20+ repositories to use this. Alexei and others have at least started the first stage of a rollout - the blueprint(s) needs additional work, planning and discussion, but his work is a good first step (reduce the duplication of code) although I am worried that the libraries and APIs / namespaces will need to change if we continue these discussions which potentially means re-doing work. If we take a step back, a rollout process might be: 1: Solidify the libraries / layout / naming conventions (blueprint) 2: Solidify the APIs exposed to consumers (blueprint) 3: Pick up on the common-client-library-2 work which is primarily a migration of common code into oslo today, into the structure defined by 1 2 So, I sort of agree: moving / collapsing code now might be premature. I do strongly agree it should stand on its own as a library rather than an oslo incubator however. We should start with a single, clean namespace / library rather than depending on oslo directly. jesse ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 9:26 AM, Justin Hammond justin.hamm...@rackspace.commailto:justin.hamm...@rackspace.com wrote: I'm not sure if it was said, but which httplib using being used (urllib3 maybe?). Also I noticed many people were talking about supporting auth properly, but are there any intentions to properly support 'noauth' (python-neutronclient, for instance, doesn't support it properly as of this writing)? Can you detail out noauth for me; and I would say the defacto httplib in python today is python-requests - urllib3 is also good but I would say from a *consumer* standpoint requests offers more in terms of usability / extensibility On 1/15/14 10:53 PM, Alexei Kornienko alexei.kornie...@gmail.commailto:alexei.kornie...@gmail.com wrote: I did notice, however, that neutronclient is conspicuously absent from the Work Items in the blueprint's Whiteboard. It will surely be added later. We already working on several things in parallel and we will add neutronclient soon. Do you need another person to make neutron client? I would love to see a bit more detail on the structure of the lib(s), the blueprint really doesn't discuss the design/organization/intended API of the libs. For example, I would hope the distinction between the various layers of a client stack don't get lost, i.e. not mixing the low-level REST API bits with the higher-level CLI parsers and decorators. Does the long-term goals include a common caching layer? Distinction between client layers won't get lost and would only be improved. My basic idea is the following: 1) Transport layer would handle all transport related stuff - HTTP, JSON encoding, auth, caching, etc. 2) Model layer (Resource classes, BaseManager, etc.) will handle data representation, validation 3) API layer will handle all project specific stuff - url mapping, etc. (This will be imported to use client in other applications) 4) Cli level will handle all stuff related to cli mapping - argparse, argcomplete, etc. I believe the current effort referenced by the blueprint is focusing on moving existing code into the incubator for reuse, to make it easier to restructure later. Alexei, do I have that correct? You are right. The first thing we do is try to make all clients look/work in similar way. After we'll continue our work with improving overall structure. 2014/1/16 Noorul Islam K M noo...@noorul.commailto:noo...@noorul.com Doug Hellmann doug.hellm...@dreamhost.commailto:doug.hellm...@dreamhost.com writes: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Solum is already using it https://review.openstack.org/#/c/58067/ I would love to watch this space. Regards, Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 01/16/2014 05:25 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:07 AM, Joe Gordon joe.gord...@gmail.com mailto:joe.gord...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller jesse.nol...@rackspace.com mailto:jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah chmo...@enovance.com mailto:chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones c...@tenshu.net mailto:c...@tenshu.net wrote: Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? that's what openstackclient is here for https://github.com/openstack/python-openstackclient After speaking with people working on OSC and looking at the code base in depth; I don't think this addresses what Chris is implying: OSC wraps the individual CLIs built by each project today, instead of the inverse: a common backend that the individual CLIs can wrap - the latter is an important distinction as currently, building a single binary install of OSC for say, Windows is difficult given the dependency tree incurred by each of the wrapped CLIs, difference in dependencies, structure, etc. Also, wrapping a series of inconsistent back end Client classes / functions / methods means that the layer that presents a consistent user interface (OSC) to the user is made more complex juggling names/renames/commands/etc. In the inverted case of what we have today (single backend); as a developer of user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to: from openstack.common.api import Auth from openstack.common.api import Compute from openstack.common.util import cli_tools my_cli = cli_tools.build(...) def my_command(cli): compute = Compute(Auth(cli.tentant..., connect=True)) compute.list_flavors() This would mean that *even if the individual clients needed or wanted to keep their specific CLIs, they would be able to use a not least common denominator back end (each service can have a rich common.api.compute.py http://common.api.compute.py/ or api.compute/client.py and extend where needed. However tools like horizon / openstackclient can choose not to leverage the power user/operator/admin components and present a simplified user interface. I'm working on a wiki page + blueprint to brainstorm how we could accomplish this based off of what work is in flight today (see doug's linked blueprint) and sussing out a layout / API strawman for discussion. Some of the additions that came out of this email threads and others: 1. Common backend should provide / offer caching utilities 2. Auth retries need to be handled by the auth object, and each sub-project delegates to the auth object to manage that. 3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing I am happy to see this work being done, there is definitely a lot of work to be done on the clients. This blueprint sounds like its still being fleshed out, so I am wondering what the value is of the current patches https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z Those patches mainly sync cliutils and apiutils from oslo into the assorted clients. But if this blueprint is about the python API and not the CLI (as that would be the openstack-pythonclient), why sync in apiutils? Also does this need to go through oslo-incubator or can this start out as a library? Making this a library earlier on will reduce the number of patches needed to get 20+ repositories to use this. Alexei and others have at least started the first stage of a rollout - the blueprint(s) needs additional work, planning and discussion, but his work is a good first step (reduce the duplication of code) although I am worried that the libraries and APIs / namespaces will need to change if we continue these discussions which potentially means re-doing work. If we take a step back, a rollout process might be: 1: Solidify the libraries / layout / naming conventions (blueprint) 2: Solidify the APIs exposed to consumers (blueprint) 3: Pick up on the common-client-library-2 work which is primarily a migration of common code into oslo today, into the structure defined by 1 2 So, I sort of agree: moving / collapsing code now might be premature. I do strongly agree it should stand on its own as a library rather than an oslo incubator however. We should start with a single, clean namespace / library rather than depending on oslo directly. Knowing usual openstack workflow I'm afraid that #1,#2 with a waterfall approach may take years to be complete. And after they'll be approved it will become clear that this architecture is already outdated. We try to use iterative approach for
Re: [openstack-dev] a common client library
On Thu, 2014-01-16 at 09:03 +0100, Flavio Percoco wrote: On 15/01/14 21:35 +, Jesse Noller wrote: On Jan 15, 2014, at 1:37 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 *raises hand* Me me! I’ve been talking to many contributors about the Developer Experience stuff I emailed out prior to the holidays and I was starting blueprint work, but this is a great pointer. I’m going to have to sync up with Alexei. I think solving this for openstack developers and maintainers as the blueprint says is a big win in terms of code reuse / maintenance and consistent but more so for *end-user developers* consuming openstack clouds. Some background - there’s some terminology mismatch but the rough idea is the same: * A centralized “SDK” (Software Development Kit) would be built condensing the common code and logic and operations into a single namespace. * This SDK would be able to be used by “downstream” CLIs - essentially the CLIs become a specialized front end - and in some cases, only an argparse or cliff front-end to the SDK methods located in the (for example) openstack.client.api.compute * The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived clients - all of the openstack.client.api.** classes would accept an Auth object to delegate management / mocking of the Auth / service catalog stuff to. This means developers building applications (say for example, horizon) don’t need to worry about token/expired authentication/etc. * Simplify the dependency graph code for the existing tools to enable single binary installs (py2exe, py2app, etc) for end users of the command line tools. Short version: if a developer wants to consume an openstack cloud; the would have a single SDK with minimal dependencies and import from a single namespace. An example application might look like: from openstack.api import AuthV2 from openstack.api import ComputeV2 myauth = AuthV2(…., connect=True) compute = ComputeV2(myauth) compute.list_flavors() I know this is an example but, could we leave the version out of the class name? Having something like: from openstack.api.v2 import Compute or from openstack.compute.v2 import Instance (just made that up) for marconi we're using the later. ++ -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 4:37 PM, Jesse Noller jesse.nol...@rackspace.comwrote: Can you detail out noauth for me; and I would say the defacto httplib in python today is python-requests - urllib3 is also good but I would say from a *consumer* standpoint requests offers more in terms of usability / extensibility FYI: python-requests is using urllib3 for its backend (which use httplib after that). Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 9:37 AM, Jesse Noller jesse.nol...@rackspace.comwrote: On Jan 16, 2014, at 9:26 AM, Justin Hammond justin.hamm...@rackspace.com wrote: I'm not sure if it was said, but which httplib using being used (urllib3 maybe?). Also I noticed many people were talking about supporting auth properly, but are there any intentions to properly support 'noauth' (python-neutronclient, for instance, doesn't support it properly as of this writing)? Can you detail out noauth for me; and I would say the defacto httplib in python today is python-requests - urllib3 is also good but I would say from a *consumer* standpoint requests offers more in terms of usability / extensibility requests is built on top of urllib3 so there's that... The biggest reaon I favor using Jamie Lennox's new session layer stuff in keystoneclient is that it better reflects the requests API instead of it being stuffed in after the fact. And as the one responsible for that stuffing, it was pretty blunt and really needs to be cleaned up more than Alessio did. only a few libs (maybe just glance and swift?) don't use requests at this point and I think the resistance there is the chunked transfers they both do. I'm really curious what 'noauth' means against APIs that have few, if any, calls that operate without a valid token. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
I'm not sure if it was said, but which httplib using being used (urllib3 maybe?). Also I noticed many people were talking about supporting auth properly, but are there any intentions to properly support 'noauth' (python-neutronclient, for instance, doesn't support it properly as of this writing)? On 1/15/14 10:53 PM, Alexei Kornienko alexei.kornie...@gmail.com wrote: I did notice, however, that neutronclient is conspicuously absent from the Work Items in the blueprint's Whiteboard. It will surely be added later. We already working on several things in parallel and we will add neutronclient soon. Do you need another person to make neutron client? I would love to see a bit more detail on the structure of the lib(s), the blueprint really doesn't discuss the design/organization/intended API of the libs. For example, I would hope the distinction between the various layers of a client stack don't get lost, i.e. not mixing the low-level REST API bits with the higher-level CLI parsers and decorators. Does the long-term goals include a common caching layer? Distinction between client layers won't get lost and would only be improved. My basic idea is the following: 1) Transport layer would handle all transport related stuff - HTTP, JSON encoding, auth, caching, etc. 2) Model layer (Resource classes, BaseManager, etc.) will handle data representation, validation 3) API layer will handle all project specific stuff - url mapping, etc. (This will be imported to use client in other applications) 4) Cli level will handle all stuff related to cli mapping - argparse, argcomplete, etc. I believe the current effort referenced by the blueprint is focusing on moving existing code into the incubator for reuse, to make it easier to restructure later. Alexei, do I have that correct? You are right. The first thing we do is try to make all clients look/work in similar way. After we'll continue our work with improving overall structure. 2014/1/16 Noorul Islam K M noo...@noorul.com Doug Hellmann doug.hellm...@dreamhost.com writes: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Solum is already using it https://review.openstack.org/#/c/58067/ I would love to watch this space. Regards, Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 8:45 AM, Jesse Noller jesse.nol...@rackspace.comwrote: After speaking with people working on OSC and looking at the code base in depth; I don’t think this addresses what Chris is implying: OSC wraps the individual CLIs built by each project today, instead of the inverse: a common backend that the individual CLIs can wrap - the latter is an important distinction as currently, building a single binary install of OSC for say, Windows is difficult given the dependency tree incurred by each of the wrapped CLIs, difference in dependencies, structure, etc. OSC is the top of the cake and was the most beneficial to user experience so it went first. I would love to consume fewer libraries and dependencies, so much that I still have a project to do this in a language that can easily ship a single binary client. What I think we're talking about here is the bottom of the cake and eliminating duplication and accumulated cruft from repeated forking and later direction changes. The creamy gooey middle is the API-specific bits that right stay exactly where they are (for now??) and can continue to ship a stand-alone cli if they wish. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Wed, Jan 15, 2014 at 4:35 PM, Jesse Noller jesse.nol...@rackspace.comwrote: On Jan 15, 2014, at 1:37 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 *raises hand* Me me! I’ve been talking to many contributors about the Developer Experience stuff I emailed out prior to the holidays and I was starting blueprint work, but this is a great pointer. I’m going to have to sync up with Alexei. I think solving this for openstack developers and maintainers as the blueprint says is a big win in terms of code reuse / maintenance and consistent but more so for *end-user developers* consuming openstack clouds. Some background - there’s some terminology mismatch but the rough idea is the same: * A centralized “SDK” (Software Development Kit) would be built condensing the common code and logic and operations into a single namespace. * This SDK would be able to be used by “downstream” CLIs - essentially the CLIs become a specialized front end - and in some cases, only an argparse or cliff front-end to the SDK methods located in the (for example) openstack.client.api.compute * The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived clients - all of the openstack.client.api.** classes would accept an Auth object to delegate management / mocking of the Auth / service catalog stuff to. This means developers building applications (say for example, horizon) don’t need to worry about token/expired authentication/etc. * Simplify the dependency graph code for the existing tools to enable single binary installs (py2exe, py2app, etc) for end users of the command line tools. Short version: if a developer wants to consume an openstack cloud; the would have a single SDK with minimal dependencies and import from a single namespace. An example application might look like: from openstack.api import AuthV2 from openstack.api import ComputeV2 myauth = AuthV2(…., connect=True) compute = ComputeV2(myauth) compute.list_flavors() This greatly improves the developer experience both internal to openstack and externally. Currently OpenStack has 22+ (counting stackforge) potential libraries a developer may need to install to use a full deployment of OpenStack: * python-keystoneclient (identity) * python-glanceclient (image) * python-novaclient (compute) * python-troveclient (database) * python-neutronclient (network) * python-ironicclient (bare metal) * python-heatclient (orchestration) * python-cinderclient (block storage) * python-ceilometerclient (telemetry, metrics billing) * python-swiftclient (object storage) * python-savannaclient (big data) * python-openstackclient (meta client package) * python-marconiclient (queueing) * python-tuskarclient (tripleo / management) * python-melangeclient (dead) * python-barbicanclient (secrets) * python-solumclient (ALM) * python-muranoclient (application catalog) * python-manilaclient (shared filesystems) * python-libraclient (load balancers) * python-climateclient (reservations) * python-designateclient (Moniker/DNS) If you exclude the above and look on PyPI: On PyPi (client libraries/SDKs only, excluding the above - not maintained by openstack): * hpcloud-auth-openstack 1.0 * python-openstackclient 0.2.2 * rackspace-auth-openstack 1.1 * posthaste 0.2.2 * pyrax 1.6.2 * serverherald 0.0.1 * warm 0.3.1 * vaporize 0.3.2 * swiftsc (https://github.com/mkouhei/swiftsc) * bookofnova 0.007 * nova-adminclient 0.1.8 * python-quantumclient 2.2.4.3 * python-stackhelper 0.0.7.1.gcab1eb0 * swift-bench 1.0 * swiftly 1.12 * txAWS 0.2.3 * cfupload 0.5.1 * python-reddwarfclient 0.1.2 * python-automationclient 1.2.1 * rackspace-glanceclient 0.9 * rackspace-novaclient 1.4 If you ignore PyPI and just want to install the base say - 7 services, each one of the python-** clients has its own dependency tree and may or may not build from one of the others. If a vendor wants to extend any of them, it’s basically a fork instead of a clean plugin system. On the CLI front - this would *greatly* simplify the work openstackclient needs to do - it would be able to import from the main SDK and simply provide the noun-verb command line
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 9:54 AM, Alexei Kornienko alexei.kornie...@gmail.commailto:alexei.kornie...@gmail.com wrote: On 01/16/2014 05:25 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:07 AM, Joe Gordon joe.gord...@gmail.commailto:joe.gord...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller jesse.nol...@rackspace.commailto:jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah chmo...@enovance.commailto:chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones c...@tenshu.netmailto:c...@tenshu.net wrote: Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? that's what openstackclient is here for https://github.com/openstack/python-openstackclient After speaking with people working on OSC and looking at the code base in depth; I don’t think this addresses what Chris is implying: OSC wraps the individual CLIs built by each project today, instead of the inverse: a common backend that the individual CLIs can wrap - the latter is an important distinction as currently, building a single binary install of OSC for say, Windows is difficult given the dependency tree incurred by each of the wrapped CLIs, difference in dependencies, structure, etc. Also, wrapping a series of inconsistent back end Client classes / functions / methods means that the layer that presents a consistent user interface (OSC) to the user is made more complex juggling names/renames/commands/etc. In the inverted case of what we have today (single backend); as a developer of user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to: from openstack.common.api import Auth from openstack.common.api import Compute from openstack.common.util import cli_tools my_cli = cli_tools.build(…) def my_command(cli): compute = Compute(Auth(cli.tentant…, connect=True)) compute.list_flavors() This would mean that *even if the individual clients needed or wanted to keep their specific CLIs, they would be able to use a not “least common denominator” back end (each service can have a rich common.api.compute.pyhttp://common.api.compute.py/ or api.compute/client.py and extend where needed. However tools like horizon / openstackclient can choose not to leverage the “power user/operator/admin” components and present a simplified user interface. I’m working on a wiki page + blueprint to brainstorm how we could accomplish this based off of what work is in flight today (see doug’s linked blueprint) and sussing out a layout / API strawman for discussion. Some of the additions that came out of this email threads and others: 1. Common backend should provide / offer caching utilities 2. Auth retries need to be handled by the auth object, and each sub-project delegates to the auth object to manage that. 3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing I am happy to see this work being done, there is definitely a lot of work to be done on the clients. This blueprint sounds like its still being fleshed out, so I am wondering what the value is of the current patches https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z Those patches mainly sync cliutils and apiutils from oslo into the assorted clients. But if this blueprint is about the python API and not the CLI (as that would be the openstack-pythonclient), why sync in apiutils? Also does this need to go through oslo-incubator or can this start out as a library? Making this a library earlier on will reduce the number of patches needed to get 20+ repositories to use this. Alexei and others have at least started the first stage of a rollout - the blueprint(s) needs additional work, planning and discussion, but his work is a good first step (reduce the duplication of code) although I am worried that the libraries and APIs / namespaces will need to change if we continue these discussions which potentially means re-doing work. If we take a step back, a rollout process might be: 1: Solidify the libraries / layout / naming conventions (blueprint) 2: Solidify the APIs exposed to consumers (blueprint) 3: Pick up on the common-client-library-2 work which is primarily a migration of common code into oslo today, into the structure defined by 1 2 So, I sort of agree: moving / collapsing code now might be premature. I do strongly agree it should stand on its own as a library rather than an oslo incubator however. We should start with a single, clean namespace / library rather than depending on oslo directly. Knowing usual openstack workflow I'm afraid that #1,#2 with a waterfall approach may take years to be complete. And after they'll be approved it will become clear that this architecture is already outdated. We try to use iterative approach for clients refactoring. We started our work from removing code duplication because it already gives
Re: [openstack-dev] a common client library
My prioritization of noauth is rooted in the fact that we're finding that the current pattern of hitting auth to validate a token is not scaling well. Out current solution to this scale issue is: - use noauth when possible between the services - use normal auth for public services - provide a method to create a 'trusted environment' While this problem may not be prevalent in other deployments I will add that support noauth in the client 'just makes sense' when the services themselves support them. For instance our setup looks like: User - Auth to Nova - Nova/Computes - NoAuth to neutron in 'trusted environment' It saves quite a few calls to identity in this way and scales a lot better. On 1/16/14 11:06 AM, Dean Troyer dtro...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:37 AM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 9:26 AM, Justin Hammond justin.hamm...@rackspace.com wrote: I'm not sure if it was said, but which httplib using being used (urllib3 maybe?). Also I noticed many people were talking about supporting auth properly, but are there any intentions to properly support 'noauth' (python-neutronclient, for instance, doesn't support it properly as of this writing)? Can you detail out noauth for me; and I would say the defacto httplib in python today is python-requests - urllib3 is also good but I would say from a *consumer* standpoint requests offers more in terms of usability / extensibility requests is built on top of urllib3 so there's that... The biggest reaon I favor using Jamie Lennox's new session layer stuff in keystoneclient is that it better reflects the requests API instead of it being stuffed in after the fact. And as the one responsible for that stuffing, it was pretty blunt and really needs to be cleaned up more than Alessio did. only a few libs (maybe just glance and swift?) don't use requests at this point and I think the resistance there is the chunked transfers they both do. I'm really curious what 'noauth' means against APIs that have few, if any, calls that operate without a valid token. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 9:54 AM, Alexei Kornienko alexei.kornie...@gmail.com wrote: Knowing usual openstack workflow I'm afraid that #1,#2 with a waterfall approach may take years to be complete. And after they'll be approved it will become clear that this architecture is already outdated. We try to use iterative approach for clients refactoring. We started our work from removing code duplication because it already gives a direct positive effect on client projects. If you can show us better way of moving forward please help us by uploading patches on this topic. Talk is cheap. Show me the code. (c) Linus python-keystoneclient has the Session bits commiited, and the auth bits in flight. So we'll all be using it one way or another. OSC has a very-slimmed-down model for replacing the REST and API layers already present underneath the object-store commands as an experiment to see how slim I could make then and still be usable. It isn't necessarily what I want to ship everywhere, but the REST layer is eerily similar to Jamie's in keystoneclient. There's why my bias... dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, 2014-01-16 at 10:06 -0600, Dean Troyer wrote: On Thu, Jan 16, 2014 at 9:37 AM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 9:26 AM, Justin Hammond justin.hamm...@rackspace.com wrote: I'm not sure if it was said, but which httplib using being used (urllib3 maybe?). Also I noticed many people were talking about supporting auth properly, but are there any intentions to properly support 'noauth' (python-neutronclient, for instance, doesn't support it properly as of this writing)? Can you detail out noauth for me; and I would say the defacto httplib in python today is python-requests - urllib3 is also good but I would say from a *consumer* standpoint requests offers more in terms of usability / extensibility requests is built on top of urllib3 so there's that... The biggest reaon I favor using Jamie Lennox's new session layer stuff in keystoneclient is that it better reflects the requests API instead of it being stuffed in after the fact. And as the one responsible for that stuffing, it was pretty blunt and really needs to be cleaned up more than Alessio did. only a few libs (maybe just glance and swift?) don't use requests at this point and I think the resistance there is the chunked transfers they both do. Right, but requests supports chunked-transfer encoding properly, so really there's no reason those clients could not move to a requests-based codebase. Best, -jay I'm really curious what 'noauth' means against APIs that have few, if any, calls that operate without a valid token. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 5:23 PM, Jay Pipes jaypi...@gmail.com wrote: Right, but requests supports chunked-transfer encoding properly, so really there's no reason those clients could not move to a requests-based codebase. We had that discussion for swiftclient and we are not against it but unfortunately there is no way to disable compression using requests which is a requirement for performances with swift (and security). see: https://review.openstack.org/#/c/33473/ and: https://github.com/kennethreitz/requests/issues/1853 Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 10:25 AM, Jesse Noller jesse.nol...@rackspace.comwrote: On Jan 16, 2014, at 9:07 AM, Joe Gordon joe.gord...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller jesse.nol...@rackspace.comwrote: On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones c...@tenshu.net wrote: Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? that's what openstackclient is here for https://github.com/openstack/python-openstackclient After speaking with people working on OSC and looking at the code base in depth; I don’t think this addresses what Chris is implying: OSC wraps the individual CLIs built by each project today, instead of the inverse: a common backend that the individual CLIs can wrap - the latter is an important distinction as currently, building a single binary install of OSC for say, Windows is difficult given the dependency tree incurred by each of the wrapped CLIs, difference in dependencies, structure, etc. Also, wrapping a series of inconsistent back end Client classes / functions / methods means that the layer that presents a consistent user interface (OSC) to the user is made more complex juggling names/renames/commands/etc. In the inverted case of what we have today (single backend); as a developer of user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to: from openstack.common.api import Auth from openstack.common.api import Compute from openstack.common.util import cli_tools my_cli = cli_tools.build(…) def my_command(cli): compute = Compute(Auth(cli.tentant…, connect=True)) compute.list_flavors() This would mean that *even if the individual clients needed or wanted to keep their specific CLIs, they would be able to use a not “least common denominator” back end (each service can have a rich common.api.compute.pyor api.compute/client.py and extend where needed. However tools like horizon / openstackclient can choose not to leverage the “power user/operator/admin” components and present a simplified user interface. I’m working on a wiki page + blueprint to brainstorm how we could accomplish this based off of what work is in flight today (see doug’s linked blueprint) and sussing out a layout / API strawman for discussion. Some of the additions that came out of this email threads and others: 1. Common backend should provide / offer caching utilities 2. Auth retries need to be handled by the auth object, and each sub-project delegates to the auth object to manage that. 3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing I am happy to see this work being done, there is definitely a lot of work to be done on the clients. This blueprint sounds like its still being fleshed out, so I am wondering what the value is of the current patches https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z Those patches mainly sync cliutils and apiutils from oslo into the assorted clients. But if this blueprint is about the python API and not the CLI (as that would be the openstack-pythonclient), why sync in apiutils? Also does this need to go through oslo-incubator or can this start out as a library? Making this a library earlier on will reduce the number of patches needed to get 20+ repositories to use this. Alexei and others have at least started the first stage of a rollout - the blueprint(s) needs additional work, planning and discussion, but his work is a good first step (reduce the duplication of code) although I am worried that the libraries and APIs / namespaces will need to change if we continue these discussions which potentially means re-doing work. If we take a step back, a rollout process might be: 1: Solidify the libraries / layout / naming conventions (blueprint) 2: Solidify the APIs exposed to consumers (blueprint) 3: Pick up on the common-client-library-2 work which is primarily a migration of common code into oslo today, into the structure defined by 1 2 So, I sort of agree: moving / collapsing code now might be premature. I do strongly agree it should stand on its own as a library rather than an oslo incubator however. We should start with a single, clean namespace / library rather than depending on oslo directly. Sounds good to me, thanks. jesse ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 01/16/2014 06:15 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:54 AM, Alexei Kornienko alexei.kornie...@gmail.com mailto:alexei.kornie...@gmail.com wrote: On 01/16/2014 05:25 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:07 AM, Joe Gordon joe.gord...@gmail.com mailto:joe.gord...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller jesse.nol...@rackspace.com mailto:jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah chmo...@enovance.com mailto:chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones c...@tenshu.net mailto:c...@tenshu.net wrote: Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? that's what openstackclient is here for https://github.com/openstack/python-openstackclient After speaking with people working on OSC and looking at the code base in depth; I don't think this addresses what Chris is implying: OSC wraps the individual CLIs built by each project today, instead of the inverse: a common backend that the individual CLIs can wrap - the latter is an important distinction as currently, building a single binary install of OSC for say, Windows is difficult given the dependency tree incurred by each of the wrapped CLIs, difference in dependencies, structure, etc. Also, wrapping a series of inconsistent back end Client classes / functions / methods means that the layer that presents a consistent user interface (OSC) to the user is made more complex juggling names/renames/commands/etc. In the inverted case of what we have today (single backend); as a developer of user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to: from openstack.common.api import Auth from openstack.common.api import Compute from openstack.common.util import cli_tools my_cli = cli_tools.build(...) def my_command(cli): compute = Compute(Auth(cli.tentant..., connect=True)) compute.list_flavors() This would mean that *even if the individual clients needed or wanted to keep their specific CLIs, they would be able to use a not least common denominator back end (each service can have a rich common.api.compute.py http://common.api.compute.py/ or api.compute/client.py and extend where needed. However tools like horizon / openstackclient can choose not to leverage the power user/operator/admin components and present a simplified user interface. I'm working on a wiki page + blueprint to brainstorm how we could accomplish this based off of what work is in flight today (see doug's linked blueprint) and sussing out a layout / API strawman for discussion. Some of the additions that came out of this email threads and others: 1. Common backend should provide / offer caching utilities 2. Auth retries need to be handled by the auth object, and each sub-project delegates to the auth object to manage that. 3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing I am happy to see this work being done, there is definitely a lot of work to be done on the clients. This blueprint sounds like its still being fleshed out, so I am wondering what the value is of the current patches https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z Those patches mainly sync cliutils and apiutils from oslo into the assorted clients. But if this blueprint is about the python API and not the CLI (as that would be the openstack-pythonclient), why sync in apiutils? Also does this need to go through oslo-incubator or can this start out as a library? Making this a library earlier on will reduce the number of patches needed to get 20+ repositories to use this. Alexei and others have at least started the first stage of a rollout - the blueprint(s) needs additional work, planning and discussion, but his work is a good first step (reduce the duplication of code) although I am worried that the libraries and APIs / namespaces will need to change if we continue these discussions which potentially means re-doing work. If we take a step back, a rollout process might be: 1: Solidify the libraries / layout / naming conventions (blueprint) 2: Solidify the APIs exposed to consumers (blueprint) 3: Pick up on the common-client-library-2 work which is primarily a migration of common code into oslo today, into the structure defined by 1 2 So, I sort of agree: moving / collapsing code now might be premature. I do strongly agree it should stand on its own as a library rather than an oslo incubator however. We should start with a single, clean namespace / library rather than depending on oslo directly. Knowing usual openstack workflow I'm afraid that #1,#2 with a waterfall
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 10:23 AM, Jay Pipes jaypi...@gmail.com wrote: Right, but requests supports chunked-transfer encoding properly, so really there's no reason those clients could not move to a requests-based codebase. Absolutely...it was totally me chickening out at the time why they didn't get changed. I feel a bit braver now... :) dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 8:06 AM, Dean Troyer dtro...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:37 AM, Jesse Noller jesse.nol...@rackspace.comwrote: On Jan 16, 2014, at 9:26 AM, Justin Hammond justin.hamm...@rackspace.com wrote: I'm not sure if it was said, but which httplib using being used (urllib3 maybe?). Also I noticed many people were talking about supporting auth properly, but are there any intentions to properly support 'noauth' (python-neutronclient, for instance, doesn't support it properly as of this writing)? Can you detail out noauth for me; and I would say the defacto httplib in python today is python-requests - urllib3 is also good but I would say from a *consumer* standpoint requests offers more in terms of usability / extensibility requests is built on top of urllib3 so there's that... The biggest reaon I favor using Jamie Lennox's new session layer stuff in keystoneclient is that it better reflects the requests API instead of it being stuffed in after the fact. And as the one responsible for that stuffing, it was pretty blunt and really needs to be cleaned up more than Alessio did. only a few libs (maybe just glance and swift?) don't use requests at this point and I think the resistance there is the chunked transfers they both do. There's a few more items here that are needed for glance to be able to work with requests (which we really really want). 1) Support for 100-expect-continue is probably going to be required in glance as well as swift 2) Support for turning off tls/ssl compression (our streams are already compressed) I feel like we *must* have somebody here who is able and willing to add these features to requests, which seems like the right approach. I'm really curious what 'noauth' means against APIs that have few, if any, calls that operate without a valid token. dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Wed, Jan 15, 2014 at 7:53 PM, Alexei Kornienko alexei.kornie...@gmail.com wrote: I did notice, however, that neutronclient is conspicuously absent from the Work Items in the blueprint's Whiteboard. It will surely be added later. We already working on several things in parallel and we will add neutronclient soon. I would love to see a bit more detail on the structure of the lib(s), the blueprint really doesn't discuss the design/organization/intended API of the libs. For example, I would hope the distinction between the various layers of a client stack don't get lost, i.e. not mixing the low-level REST API bits with the higher-level CLI parsers and decorators. Does the long-term goals include a common caching layer? Distinction between client layers won't get lost and would only be improved. My basic idea is the following: 1) Transport layer would handle all transport related stuff - HTTP, JSON encoding, auth, caching, etc. 2) Model layer (Resource classes, BaseManager, etc.) will handle data representation, validation 3) API layer will handle all project specific stuff - url mapping, etc. (This will be imported to use client in other applications) 4) Cli level will handle all stuff related to cli mapping - argparse, argcomplete, etc. I'm really excited about this. I think consolidating layers 1 and 4 will be a huge benefit for deployers and users. I'm hoping we can structure layers 2 and 3 a bit flexibly to allow for existing project differences and proper ownership. For example, in Glance we use jsonschema somewhat so our validation is a bit different. Also, I consider the definition of resources and url mappings for images to be something that should be owned by the Images program. I'm confident, however, that we can figure out how to structure the libraries, deliverables, and process to reflect that ownership. I believe the current effort referenced by the blueprint is focusing on moving existing code into the incubator for reuse, to make it easier to restructure later. Alexei, do I have that correct? You are right. The first thing we do is try to make all clients look/work in similar way. After we'll continue our work with improving overall structure. 2014/1/16 Noorul Islam K M noo...@noorul.com Doug Hellmann doug.hellm...@dreamhost.com writes: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Solum is already using it https://review.openstack.org/#/c/58067/ I would love to watch this space. Regards, Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 12:03 AM, Flavio Percoco fla...@redhat.com wrote: On 15/01/14 21:35 +, Jesse Noller wrote: On Jan 15, 2014, at 1:37 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 *raises hand* Me me! I’ve been talking to many contributors about the Developer Experience stuff I emailed out prior to the holidays and I was starting blueprint work, but this is a great pointer. I’m going to have to sync up with Alexei. I think solving this for openstack developers and maintainers as the blueprint says is a big win in terms of code reuse / maintenance and consistent but more so for *end-user developers* consuming openstack clouds. Some background - there’s some terminology mismatch but the rough idea is the same: * A centralized “SDK” (Software Development Kit) would be built condensing the common code and logic and operations into a single namespace. * This SDK would be able to be used by “downstream” CLIs - essentially the CLIs become a specialized front end - and in some cases, only an argparse or cliff front-end to the SDK methods located in the (for example) openstack.client.api.compute * The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived clients - all of the openstack.client.api.** classes would accept an Auth object to delegate management / mocking of the Auth / service catalog stuff to. This means developers building applications (say for example, horizon) don’t need to worry about token/expired authentication/etc. * Simplify the dependency graph code for the existing tools to enable single binary installs (py2exe, py2app, etc) for end users of the command line tools. Short version: if a developer wants to consume an openstack cloud; the would have a single SDK with minimal dependencies and import from a single namespace. An example application might look like: from openstack.api import AuthV2 from openstack.api import ComputeV2 myauth = AuthV2(…., connect=True) compute = ComputeV2(myauth) compute.list_flavors() I know this is an example but, could we leave the version out of the class name? Having something like: from openstack.api.v2 import Compute or from openstack.compute.v2 import Instance (just made that up) for marconi we're using the later. Just throwing this out there because it seems relevant to client design. As we've been looking at porting clients to using v2 of the Images API, its seems more and more to me that including the *server* version in the main import path is a real obstacle. IMO any future client libs should write library interfaces based on the peculiarities of user needs, not based on the vagaries of the server version. So as a user of this library I would do something like: 1 from openstack.api import images 2 client = images.make_me_a_client(auth_url, etcetera) # all version negotiation is happening here 3 client.list_images() # works more or less same no matter who I'm talking to Now, there would still likely be hidden implementation code that is different per server version and which is instantiated in line 2 above, and maybe that's the library path stuff you are talking about. This greatly improves the developer experience both internal to openstack and externally. Currently OpenStack has 22+ (counting stackforge) potential libraries a developer may need to install to use a full deployment of OpenStack: * python-keystoneclient (identity) * python-glanceclient (image) * python-novaclient (compute) * python-troveclient (database) * python-neutronclient (network) * python-ironicclient (bare metal) * python-heatclient (orchestration) * python-cinderclient (block storage) * python-ceilometerclient (telemetry, metrics billing) * python-swiftclient (object storage) * python-savannaclient (big data) * python-openstackclient (meta client package) * python-marconiclient (queueing) * python-tuskarclient (tripleo / management) * python-melangeclient (dead) * python-barbicanclient (secrets) * python-solumclient (ALM) * python-muranoclient (application catalog) * python-manilaclient (shared filesystems) * python-libraclient (load balancers) * python-climateclient (reservations) * python-designateclient
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 12:07 PM, Alexei Kornienko alexei.kornie...@gmail.com wrote: On 01/16/2014 06:15 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:54 AM, Alexei Kornienko alexei.kornie...@gmail.com wrote: On 01/16/2014 05:25 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:07 AM, Joe Gordon joe.gord...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller jesse.nol...@rackspace.comwrote: On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones c...@tenshu.net wrote: Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? that's what openstackclient is here for https://github.com/openstack/python-openstackclient After speaking with people working on OSC and looking at the code base in depth; I don’t think this addresses what Chris is implying: OSC wraps the individual CLIs built by each project today, instead of the inverse: a common backend that the individual CLIs can wrap - the latter is an important distinction as currently, building a single binary install of OSC for say, Windows is difficult given the dependency tree incurred by each of the wrapped CLIs, difference in dependencies, structure, etc. Also, wrapping a series of inconsistent back end Client classes / functions / methods means that the layer that presents a consistent user interface (OSC) to the user is made more complex juggling names/renames/commands/etc. In the inverted case of what we have today (single backend); as a developer of user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to: from openstack.common.api import Auth from openstack.common.api import Compute from openstack.common.util import cli_tools my_cli = cli_tools.build(…) def my_command(cli): compute = Compute(Auth(cli.tentant…, connect=True)) compute.list_flavors() This would mean that *even if the individual clients needed or wanted to keep their specific CLIs, they would be able to use a not “least common denominator” back end (each service can have a rich common.api.compute.pyor api.compute/client.py and extend where needed. However tools like horizon / openstackclient can choose not to leverage the “power user/operator/admin” components and present a simplified user interface. I’m working on a wiki page + blueprint to brainstorm how we could accomplish this based off of what work is in flight today (see doug’s linked blueprint) and sussing out a layout / API strawman for discussion. Some of the additions that came out of this email threads and others: 1. Common backend should provide / offer caching utilities 2. Auth retries need to be handled by the auth object, and each sub-project delegates to the auth object to manage that. 3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing I am happy to see this work being done, there is definitely a lot of work to be done on the clients. This blueprint sounds like its still being fleshed out, so I am wondering what the value is of the current patches https://review.openstack.org/#/q/topic:bp/common-client-library-2,n,z Those patches mainly sync cliutils and apiutils from oslo into the assorted clients. But if this blueprint is about the python API and not the CLI (as that would be the openstack-pythonclient), why sync in apiutils? Also does this need to go through oslo-incubator or can this start out as a library? Making this a library earlier on will reduce the number of patches needed to get 20+ repositories to use this. Alexei and others have at least started the first stage of a rollout - the blueprint(s) needs additional work, planning and discussion, but his work is a good first step (reduce the duplication of code) although I am worried that the libraries and APIs / namespaces will need to change if we continue these discussions which potentially means re-doing work. If we take a step back, a rollout process might be: 1: Solidify the libraries / layout / naming conventions (blueprint) 2: Solidify the APIs exposed to consumers (blueprint) 3: Pick up on the common-client-library-2 work which is primarily a migration of common code into oslo today, into the structure defined by 1 2 So, I sort of agree: moving / collapsing code now might be premature. I do strongly agree it should stand on its own as a library rather than an oslo incubator however. We should start with a single, clean namespace / library rather than depending on oslo directly. Knowing usual openstack workflow I'm afraid that #1,#2 with a waterfall approach may take years to be complete. And after they'll be approved it will become clear that this architecture is already outdated. We try to use iterative approach for clients refactoring. We started our work from removing code
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 11:39 AM, Mark Washenberger mark.washenber...@markwash.netmailto:mark.washenber...@markwash.net wrote: On Thu, Jan 16, 2014 at 8:06 AM, Dean Troyer dtro...@gmail.commailto:dtro...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:37 AM, Jesse Noller jesse.nol...@rackspace.commailto:jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 9:26 AM, Justin Hammond justin.hamm...@rackspace.commailto:justin.hamm...@rackspace.com wrote: I'm not sure if it was said, but which httplib using being used (urllib3 maybe?). Also I noticed many people were talking about supporting auth properly, but are there any intentions to properly support 'noauth' (python-neutronclient, for instance, doesn't support it properly as of this writing)? Can you detail out noauth for me; and I would say the defacto httplib in python today is python-requests - urllib3 is also good but I would say from a *consumer* standpoint requests offers more in terms of usability / extensibility requests is built on top of urllib3 so there's that... The biggest reaon I favor using Jamie Lennox's new session layer stuff in keystoneclient is that it better reflects the requests API instead of it being stuffed in after the fact. And as the one responsible for that stuffing, it was pretty blunt and really needs to be cleaned up more than Alessio did. only a few libs (maybe just glance and swift?) don't use requests at this point and I think the resistance there is the chunked transfers they both do. There's a few more items here that are needed for glance to be able to work with requests (which we really really want). 1) Support for 100-expect-continue is probably going to be required in glance as well as swift 2) Support for turning off tls/ssl compression (our streams are already compressed) I feel like we *must* have somebody here who is able and willing to add these features to requests, which seems like the right approach. Let me talk to upstream about this; I know a lot of people involved. Patches from us probably needed, but I’ll ask I'm really curious what 'noauth' means against APIs that have few, if any, calls that operate without a valid token. dt -- Dean Troyer dtro...@gmail.commailto:dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
Hello Joe, continuous refactoring and syncing across 22+ repositories sounds like a nightmare, one that I would like to avoid. You are right this is not easy. However I have several reasons to do that: The hardest part is to bring basic stuff in sync across all projects (That's what we are doing now). Later we'll work directly with oslo lib and just sync changes from it. We could introduce a standalone library to avoid the need to sync oslo code across all projects but it brings additional problems: 1) We would have to maintain rationale versioning and backwards compatibility of this library. If we start library from scratch we'll have to add/change lots of stuff before we'll reach some stability period. 2)Another option would be to follow waterfall process and create a solid library interface before including it to all client projects. However such approach this period can take unknown amount of time and can be easily failed during integration stage cause requirements change or some other reason. Please let me know what you think. Best Regards, Alexei On 01/16/2014 08:16 PM, Joe Gordon wrote: On Thu, Jan 16, 2014 at 12:07 PM, Alexei Kornienko alexei.kornie...@gmail.com mailto:alexei.kornie...@gmail.com wrote: On 01/16/2014 06:15 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:54 AM, Alexei Kornienko alexei.kornie...@gmail.com mailto:alexei.kornie...@gmail.com wrote: On 01/16/2014 05:25 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:07 AM, Joe Gordon joe.gord...@gmail.com mailto:joe.gord...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller jesse.nol...@rackspace.com mailto:jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah chmo...@enovance.com mailto:chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones c...@tenshu.net mailto:c...@tenshu.net wrote: Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? that's what openstackclient is here for https://github.com/openstack/python-openstackclient After speaking with people working on OSC and looking at the code base in depth; I don't think this addresses what Chris is implying: OSC wraps the individual CLIs built by each project today, instead of the inverse: a common backend that the individual CLIs can wrap - the latter is an important distinction as currently, building a single binary install of OSC for say, Windows is difficult given the dependency tree incurred by each of the wrapped CLIs, difference in dependencies, structure, etc. Also, wrapping a series of inconsistent back end Client classes / functions / methods means that the layer that presents a consistent user interface (OSC) to the user is made more complex juggling names/renames/commands/etc. In the inverted case of what we have today (single backend); as a developer of user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to: from openstack.common.api import Auth from openstack.common.api import Compute from openstack.common.util import cli_tools my_cli = cli_tools.build(...) def my_command(cli): compute = Compute(Auth(cli.tentant..., connect=True)) compute.list_flavors() This would mean that *even if the individual clients needed or wanted to keep their specific CLIs, they would be able to use a not least common denominator back end (each service can have a rich common.api.compute.py http://common.api.compute.py/ or api.compute/client.py and extend where needed. However tools like horizon / openstackclient can choose not to leverage the power user/operator/admin components and present a simplified user interface. I'm working on a wiki page + blueprint to brainstorm how we could accomplish this based off of what work is in flight today (see doug's linked blueprint) and sussing out a layout / API strawman for discussion. Some of the additions that came out of this email threads and others: 1. Common backend should provide / offer caching utilities 2. Auth retries need to be handled by the auth object, and each sub-project delegates to the auth object to manage that. 3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing I am happy to see this work being done, there is definitely a lot of work to be done on the clients. This blueprint sounds like its still being fleshed out, so I am wondering what the value is of the current patches
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 2:03 PM, Alexei Kornienko alexei.kornie...@gmail.com wrote: Hello Joe, continuous refactoring and syncing across 22+ repositories sounds like a nightmare, one that I would like to avoid. You are right this is not easy. However I have several reasons to do that: The hardest part is to bring basic stuff in sync across all projects (That's what we are doing now). Later we'll work directly with oslo lib and just sync changes from it. My concern here is that is just extra churn for minimal value, assuming there are plans to completely overhaul this in the near future. Perhaps this blueprint is really two: 1) do short term work to reduplicate the common client functionality. This would cover the current patches and involve oslo-incubator, and is limited in scope and shouldn't require to many oslo sync iterations. 2) major overhaul of client libraries so they are all based off a common base library. This would cover namespace changes, and possible a push to move CLI into python-openstackclient We could introduce a standalone library to avoid the need to sync oslo code across all projects but it brings additional problems: 1) We would have to maintain rationale versioning and backwards compatibility of this library. If we start library from scratch we'll have to add/change lots of stuff before we'll reach some stability period. 2)Another option would be to follow waterfall process and create a solid library interface before including it to all client projects. However such approach this period can take unknown amount of time and can be easily failed during integration stage cause requirements change or some other reason. Please let me know what you think. Best Regards, Alexei On 01/16/2014 08:16 PM, Joe Gordon wrote: On Thu, Jan 16, 2014 at 12:07 PM, Alexei Kornienko alexei.kornie...@gmail.com wrote: On 01/16/2014 06:15 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:54 AM, Alexei Kornienko alexei.kornie...@gmail.com wrote: On 01/16/2014 05:25 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:07 AM, Joe Gordon joe.gord...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones c...@tenshu.net wrote: Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? that's what openstackclient is here for https://github.com/openstack/python-openstackclient After speaking with people working on OSC and looking at the code base in depth; I don’t think this addresses what Chris is implying: OSC wraps the individual CLIs built by each project today, instead of the inverse: a common backend that the individual CLIs can wrap - the latter is an important distinction as currently, building a single binary install of OSC for say, Windows is difficult given the dependency tree incurred by each of the wrapped CLIs, difference in dependencies, structure, etc. Also, wrapping a series of inconsistent back end Client classes / functions / methods means that the layer that presents a consistent user interface (OSC) to the user is made more complex juggling names/renames/commands/etc. In the inverted case of what we have today (single backend); as a developer of user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to: from openstack.common.api import Auth from openstack.common.api import Compute from openstack.common.util import cli_tools my_cli = cli_tools.build(…) def my_command(cli): compute = Compute(Auth(cli.tentant…, connect=True)) compute.list_flavors() This would mean that *even if the individual clients needed or wanted to keep their specific CLIs, they would be able to use a not “least common denominator” back end (each service can have a rich common.api.compute.py or api.compute/client.py and extend where needed. However tools like horizon / openstackclient can choose not to leverage the “power user/operator/admin” components and present a simplified user interface. I’m working on a wiki page + blueprint to brainstorm how we could accomplish this based off of what work is in flight today (see doug’s linked blueprint) and sussing out a layout / API strawman for discussion. Some of the additions that came out of this email threads and others: 1. Common backend should provide / offer caching utilities 2. Auth retries need to be handled by the auth object, and each sub-project delegates to the auth object to manage that. 3. Verified Mocks / Stubs / Fakes must be provided for proper unit testing I am happy to see this work being done, there is definitely a lot of work to be done on the clients. This blueprint sounds like its still being fleshed out, so I am wondering
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 2:36 PM, Joe Gordon joe.gord...@gmail.com wrote: 2) major overhaul of client libraries so they are all based off a common base library. This would cover namespace changes, and possible a push to move CLI into python-openstackclient This seems like the biggest win to me. - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
+1 to a stand alone library for this. 1) We would have to maintain rationale versioning and backwards compatibility of this library. If we start library from scratch we'll have to add/change lots of stuff before we'll reach some stability period. I don’t think this is a hard problem to solve. I think something like semantic versioning fits the bill, and allows for backwards-incompatible changes. 2)Another option would be to follow waterfall process and create a solid library interface before including it to all client projects. However such approach this period can take unknown amount of time and can be easily failed during integration stage cause requirements change or some other reason. Again, I think semver could address this. Iterate over 0.x.x series, and projects that don’t want to deal with constantly-changing code wouldn’t need to jump on board until a 1.0.0 release. -Doug M. From: Alexei Kornienko alexei.kornie...@gmail.com Reply-To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Date: Thursday, January 16, 2014 at 1:03 PM To: openstack-dev@lists.openstack.org openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] a common client library Hello Joe, continuous refactoring and syncing across 22+ repositories sounds like a nightmare, one that I would like to avoid. You are right this is not easy. However I have several reasons to do that: The hardest part is to bring basic stuff in sync across all projects (That's what we are doing now). Later we'll work directly with oslo lib and just sync changes from it. We could introduce a standalone library to avoid the need to sync oslo code across all projects but it brings additional problems: 1) We would have to maintain rationale versioning and backwards compatibility of this library. If we start library from scratch we'll have to add/change lots of stuff before we'll reach some stability period. 2)Another option would be to follow waterfall process and create a solid library interface before including it to all client projects. However such approach this period can take unknown amount of time and can be easily failed during integration stage cause requirements change or some other reason. Please let me know what you think. Best Regards, Alexei On 01/16/2014 08:16 PM, Joe Gordon wrote: On Thu, Jan 16, 2014 at 12:07 PM, Alexei Kornienko alexei.kornie...@gmail.com wrote: On 01/16/2014 06:15 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:54 AM, Alexei Kornienko alexei.kornie...@gmail.com wrote: On 01/16/2014 05:25 PM, Jesse Noller wrote: On Jan 16, 2014, at 9:07 AM, Joe Gordon joe.gord...@gmail.com wrote: On Thu, Jan 16, 2014 at 9:45 AM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 5:53 AM, Chmouel Boudjnah chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 12:38 PM, Chris Jones c...@tenshu.net wrote: Once a common library is in place, is there any intention to (or resistance against) collapsing the clients into a single project or even a single command (a la busybox)? that's what openstackclient is here for https://github.com/openstack/python-openstackclient After speaking with people working on OSC and looking at the code base in depth; I don’t think this addresses what Chris is implying: OSC wraps the individual CLIs built by each project today, instead of the inverse: a common backend that the individual CLIs can wrap - the latter is an important distinction as currently, building a single binary install of OSC for say, Windows is difficult given the dependency tree incurred by each of the wrapped CLIs, difference in dependencies, structure, etc. Also, wrapping a series of inconsistent back end Client classes / functions / methods means that the layer that presents a consistent user interface (OSC) to the user is made more complex juggling names/renames/commands/etc. In the inverted case of what we have today (single backend); as a developer of user interfaces (CLIs, Applications, Web apps (horizon)) you would be able to: from openstack.common.api import Auth from openstack.common.api import Compute from openstack.common.util import cli_tools my_cli = cli_tools.build(…) def my_command(cli): compute = Compute(Auth(cli.tentant…, connect=True)) compute.list_flavors() This would mean that *even if the individual clients needed or wanted to keep their specific CLIs, they would be able to use a not “least common denominator” back end (each service can have a rich common.api.compute.py http://common.api.compute.py/ or api.compute/client.py and extend where needed. However tools like horizon / openstackclient can choose not to leverage the “power user/operator/admin” components and present a simplified user interface. I’m working on a wiki page + blueprint to brainstorm how we could accomplish this based off of what work is in flight today (see doug’s linked
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 8:40 PM, Donald Stufft don...@stufft.io wrote: On Jan 16, 2014, at 2:36 PM, Joe Gordon joe.gord...@gmail.com wrote: 2) major overhaul of client libraries so they are all based off a common base library. This would cover namespace changes, and possible a push to move CLI into python-openstackclient This seems like the biggest win to me. +1 Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Has anyone ever considered an idea of generating a fully functional REST client automatically based on an API specification (WADL could be used for that)? Not sure how convenient it would be, it really depends on a particular implementation, but as an idea it could be at least thought of. Sounds a little bit crazy though, I recognize it :). Renat Akhmerov On 16 Jan 2014, at 11:52, Chmouel Boudjnah chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 8:40 PM, Donald Stufft don...@stufft.io wrote: On Jan 16, 2014, at 2:36 PM, Joe Gordon joe.gord...@gmail.com wrote: 2) major overhaul of client libraries so they are all based off a common base library. This would cover namespace changes, and possible a push to move CLI into python-openstackclient This seems like the biggest win to me. +1 Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Thu, Jan 16, 2014 at 2:22 PM, Renat Akhmerov rakhme...@mirantis.comwrote: - Keeping all the clients physically separate/combining them in to a single library. Two things here: - In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) FWIW, OpenStackClient has Identity built-in as very little else is useful without it. Compute, Image and Volume are in the repo but structured using the new plugin system to make sure that works well. Object-store is in the repo but I consider it experimental as it also includes the project API layer and my first cut at a common REST client. I've already written a POC for solum and some other things to demonstrate how to add additional projects simply by installing the python-*client package. https://github.com/dtroyer/python-oscplugin is a trivial example. - In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. If it is not a library that is actually shared you will get back to the current situation over time. - Has anyone ever considered an idea of generating a fully functional REST client automatically based on an API specification (WADL could be used for that)? Not sure how convenient it would be, it really depends on a particular implementation, but as an idea it could be at least thought of. Sounds a little bit crazy though, I recognize it :). When you have stable and accurate documents of this sort, let's talk. You may have noticed that few (any?) of the recent major API revs are documented in this manner... dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 4:06 PM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 2:22 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. I agree. Keeping them separate trades user usability for developer usability, I think user usability is a better thing to strive for. Has anyone ever considered an idea of generating a fully functional REST client automatically based on an API specification (WADL could be used for that)? Not sure how convenient it would be, it really depends on a particular implementation, but as an idea it could be at least thought of. Sounds a little bit crazy though, I recognize it :). Renat Akhmerov On 16 Jan 2014, at 11:52, Chmouel Boudjnah chmo...@enovance.com wrote: On Thu, Jan 16, 2014 at 8:40 PM, Donald Stufft don...@stufft.io wrote: On Jan 16, 2014, at 2:36 PM, Joe Gordon joe.gord...@gmail.com wrote: 2) major overhaul of client libraries so they are all based off a common base library. This would cover namespace changes, and possible a push to move CLI into python-openstackclient This seems like the biggest win to me. +1 Chmouel. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 3:22 PM, Renat Akhmerov rakhme...@mirantis.com wrote: Has anyone ever considered an idea of generating a fully functional REST client automatically based on an API specification (WADL could be used for that)? Not sure how convenient it would be, it really depends on a particular implementation, but as an idea it could be at least thought of. Sounds a little bit crazy though, I recognize it :). Also please no. If you want “automatic client” stop being “REST” (which has been diluted to be meaningless or at least nothing like it’s actual definition) and be REST and/or Hypermedia. Things like WADL are completely antagonistic to actual REST (aka Hypermedia). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 16 Jan 2014, at 12:36, Dean Troyer dtro...@gmail.com wrote: I've already written a POC for solum and some other things to demonstrate how to add additional projects simply by installing the python-*client package. https://github.com/dtroyer/python-oscplugin is a trivial example. Thanks, this link is helpful. In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. If it is not a library that is actually shared you will get back to the current situation over time. How is that different from any other oslo stuff? Has anyone ever considered an idea of generating a fully functional REST client automatically based on an API specification (WADL could be used for that)? Not sure how convenient it would be, it really depends on a particular implementation, but as an idea it could be at least thought of. Sounds a little bit crazy though, I recognize it :). When you have stable and accurate documents of this sort, let's talk. You may have noticed that few (any?) of the recent major API revs are documented in this manner… There’s still something to work on for us.. As far as what I’ve written, just a crazy idea that came to my mind. After all, everyone’s phone has lots of things that were considered crazy years ago. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On 16 Jan 2014, at 13:06, Jesse Noller jesse.nol...@rackspace.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. You may be right but not sure that adding another line into requirements.txt is a huge loss of usability. Renat Akhmerov___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 4:59 PM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: On 16 Jan 2014, at 13:06, Jesse Noller jesse.nol...@rackspace.commailto:jesse.nol...@rackspace.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: * Keeping all the clients physically separate/combining them in to a single library. Two things here: * In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) * In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. You may be right but not sure that adding another line into requirements.txt is a huge loss of usability. It is when that 1 dependency pulls in 6 others that pull in 10 more - every little barrier or potential failure from the inability to make a static binary to how each tool acts different is a paper cut of frustration to an end user. Most of the time the clients don't even properly install because of dependencies on setuptools plugins and other things. For developers (as I've said) the story is worse: you have potentially 22+ individual packages and their dependencies to deal with if they want to use a complete openstack install from their code. So it doesn't boil down to just 1 dependency: it's a long laundry list of things that make consumers' lives more difficult and painful. This doesn't even touch on the fact there aren't blessed SDKs or tools pointing users to consume openstack in their preferred programming language. Shipping an API isn't enough - but it can be fixed easily enough. Renat Akhmerov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 8:42 PM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 4:59 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 16 Jan 2014, at 13:06, Jesse Noller jesse.nol...@rackspace.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. You may be right but not sure that adding another line into requirements.txt is a huge loss of usability. It is when that 1 dependency pulls in 6 others that pull in 10 more - every little barrier or potential failure from the inability to make a static binary to how each tool acts different is a paper cut of frustration to an end user. Most of the time the clients don't even properly install because of dependencies on setuptools plugins and other things. For developers (as I've said) the story is worse: you have potentially 22+ individual packages and their dependencies to deal with if they want to use a complete openstack install from their code. So it doesn't boil down to just 1 dependency: it's a long laundry list of things that make consumers' lives more difficult and painful. This doesn't even touch on the fact there aren't blessed SDKs or tools pointing users to consume openstack in their preferred programming language. Shipping an API isn't enough - but it can be fixed easily enough. There’s also the discovery problem, it’s incredibly frustrating if, as I’m starting out to use an Openstack based cloud, everytime I want to touch some new segment of the service I need to go find out what the client lib is for that, possibly download the dependencies for it, possibly get it approved, etc. Splitting up services makes a lot of sense on the server side, but to the consumer a cloud often times isn’t a disjoint set of services that happen to be working in parallel, it is a single unified product where they may not know the boundary lines, or at the very least the boundaries can be fuzzy for them. Renat Akhmerov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
Ok, I think most of the reasoning you’ve said makes sense. So for a non-incubated project we’re going to have separate clients and then get them integrated into this single client once the project itself gets incubated? Or it’s going to happen when the project gets integrated into core os projects? So if yes, it’s just going to be one more incubation/integration requirement, right? Renat On 16 Jan 2014, at 18:09, Donald Stufft don...@stufft.io wrote: On Jan 16, 2014, at 8:42 PM, Jesse Noller jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 4:59 PM, Renat Akhmerov rakhme...@mirantis.com wrote: On 16 Jan 2014, at 13:06, Jesse Noller jesse.nol...@rackspace.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: Keeping all the clients physically separate/combining them in to a single library. Two things here: In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. You may be right but not sure that adding another line into requirements.txt is a huge loss of usability. It is when that 1 dependency pulls in 6 others that pull in 10 more - every little barrier or potential failure from the inability to make a static binary to how each tool acts different is a paper cut of frustration to an end user. Most of the time the clients don't even properly install because of dependencies on setuptools plugins and other things. For developers (as I've said) the story is worse: you have potentially 22+ individual packages and their dependencies to deal with if they want to use a complete openstack install from their code. So it doesn't boil down to just 1 dependency: it's a long laundry list of things that make consumers' lives more difficult and painful. This doesn't even touch on the fact there aren't blessed SDKs or tools pointing users to consume openstack in their preferred programming language. Shipping an API isn't enough - but it can be fixed easily enough. There’s also the discovery problem, it’s incredibly frustrating if, as I’m starting out to use an Openstack based cloud, everytime I want to touch some new segment of the service I need to go find out what the client lib is for that, possibly download the dependencies for it, possibly get it approved, etc. Splitting up services makes a lot of sense on the server side, but to the consumer a cloud often times isn’t a disjoint set of services that happen to be working in parallel, it is a single unified product where they may not know the boundary lines, or at the very least the boundaries can be fuzzy for them. Renat Akhmerov ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 16, 2014, at 8:36 PM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: Ok, I think most of the reasoning you’ve said makes sense. So for a non-incubated project we’re going to have separate clients and then get them integrated into this single client once the project itself gets incubated? Or it’s going to happen when the project gets integrated into core os projects? So if yes, it’s just going to be one more incubation/integration requirement, right? My gut says moving more code into Oslo and incurring that change for all 22(23 with solum) client code based short term solves one aspect of cleaning up the clients - if - the sub projects agree that the code they are now depending on presents a useful API to them. It's not clear to me that case has been explained, or the design of those new libraries sussed out. My personal opinion is that if people wanted the current consolidation to continue we let that go, but begin fleshing out what a stand alone project/library akin to horizon would look like (let's call it zenith). If we go this route we need to buildout some blueprints with design / layout / APIs for people to review and approve. When talking with dean today we actually see it as fewer layers but the most work : benefit comes from unifying and coordinating the client code from each of the projects. This latter part requires consensus from the teams on the layout and design, contributing to it, etc Dean and I started an etherpad today, I need to go back through this thread and capture more design considerations and properly capture it in the wiki. With at least a fleshed out strawman proposal for layout and design we can discuss concrete things. I think some of the contention is focused on the currently progressing work on a blueprint that doesn't fully capture the design and I'd like to avoid that. Renat On 16 Jan 2014, at 18:09, Donald Stufft don...@stufft.iomailto:don...@stufft.io wrote: On Jan 16, 2014, at 8:42 PM, Jesse Noller jesse.nol...@rackspace.commailto:jesse.nol...@rackspace.com wrote: On Jan 16, 2014, at 4:59 PM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: On 16 Jan 2014, at 13:06, Jesse Noller jesse.nol...@rackspace.commailto:jesse.nol...@rackspace.com wrote: Since it’s pretty easy to get lost among all the opinions I’d like to clarify/ask a couple of things: * Keeping all the clients physically separate/combining them in to a single library. Two things here: * In case of combining them, what exact project are we considering? If this list is limited to core projects like nova and keystone what policy could we have for other projects to join this list? (Incubation, graduation, something else?) * In terms of granularity and easiness of development I’m for keeping them separate but have them use the same boilerplate code, basically we need a OpenStack Rest Client Framework which is flexible enough to address all the needs in an abstract domain agnostic manner. I would assume that combining them would be an additional organizational burden that every stakeholder would have to deal with. Keeping them separate is awesome for *us* but really, really, really sucks for users trying to use the system. You may be right but not sure that adding another line into requirements.txt is a huge loss of usability. It is when that 1 dependency pulls in 6 others that pull in 10 more - every little barrier or potential failure from the inability to make a static binary to how each tool acts different is a paper cut of frustration to an end user. Most of the time the clients don't even properly install because of dependencies on setuptools plugins and other things. For developers (as I've said) the story is worse: you have potentially 22+ individual packages and their dependencies to deal with if they want to use a complete openstack install from their code. So it doesn't boil down to just 1 dependency: it's a long laundry list of things that make consumers' lives more difficult and painful. This doesn't even touch on the fact there aren't blessed SDKs or tools pointing users to consume openstack in their preferred programming language. Shipping an API isn't enough - but it can be fixed easily enough. There’s also the discovery problem, it’s incredibly frustrating if, as I’m starting out to use an Openstack based cloud, everytime I want to touch some new segment of the service I need to go find out what the client lib is for that, possibly download the dependencies for it, possibly get it approved, etc. Splitting up services makes a lot of sense on the server side, but to the consumer a cloud often times isn’t a disjoint set of services that happen to be working in parallel, it is a single unified product where they may not know the boundary lines, or at the very least the boundaries can be fuzzy for them. Renat Akhmerov
Re: [openstack-dev] a common client library
On Jan 15, 2014, at 1:37 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 *raises hand* Me me! I’ve been talking to many contributors about the Developer Experience stuff I emailed out prior to the holidays and I was starting blueprint work, but this is a great pointer. I’m going to have to sync up with Alexei. I think solving this for openstack developers and maintainers as the blueprint says is a big win in terms of code reuse / maintenance and consistent but more so for *end-user developers* consuming openstack clouds. Some background - there’s some terminology mismatch but the rough idea is the same: * A centralized “SDK” (Software Development Kit) would be built condensing the common code and logic and operations into a single namespace. * This SDK would be able to be used by “downstream” CLIs - essentially the CLIs become a specialized front end - and in some cases, only an argparse or cliff front-end to the SDK methods located in the (for example) openstack.client.api.compute * The SDK would handle Auth, re-auth (expired tokens, etc) for long-lived clients - all of the openstack.client.api.** classes would accept an Auth object to delegate management / mocking of the Auth / service catalog stuff to. This means developers building applications (say for example, horizon) don’t need to worry about token/expired authentication/etc. * Simplify the dependency graph code for the existing tools to enable single binary installs (py2exe, py2app, etc) for end users of the command line tools. Short version: if a developer wants to consume an openstack cloud; the would have a single SDK with minimal dependencies and import from a single namespace. An example application might look like: from openstack.api import AuthV2 from openstack.api import ComputeV2 myauth = AuthV2(…., connect=True) compute = ComputeV2(myauth) compute.list_flavors() This greatly improves the developer experience both internal to openstack and externally. Currently OpenStack has 22+ (counting stackforge) potential libraries a developer may need to install to use a full deployment of OpenStack: * python-keystoneclient (identity) * python-glanceclient (image) * python-novaclient (compute) * python-troveclient (database) * python-neutronclient (network) * python-ironicclient (bare metal) * python-heatclient (orchestration) * python-cinderclient (block storage) * python-ceilometerclient (telemetry, metrics billing) * python-swiftclient (object storage) * python-savannaclient (big data) * python-openstackclient (meta client package) * python-marconiclient (queueing) * python-tuskarclient (tripleo / management) * python-melangeclient (dead) * python-barbicanclient (secrets) * python-solumclient (ALM) * python-muranoclient (application catalog) * python-manilaclient (shared filesystems) * python-libraclient (load balancers) * python-climateclient (reservations) * python-designateclient (Moniker/DNS) If you exclude the above and look on PyPI: On PyPi (client libraries/SDKs only, excluding the above - not maintained by openstack): * hpcloud-auth-openstack 1.0 * python-openstackclient 0.2.2 * rackspace-auth-openstack 1.1 * posthaste 0.2.2 * pyrax 1.6.2 * serverherald 0.0.1 * warm 0.3.1 * vaporize 0.3.2 * swiftsc (https://github.com/mkouhei/swiftsc) * bookofnova 0.007 * nova-adminclient 0.1.8 * python-quantumclient 2.2.4.3 * python-stackhelper 0.0.7.1.gcab1eb0 * swift-bench 1.0 * swiftly 1.12 * txAWS 0.2.3 * cfupload 0.5.1 * python-reddwarfclient 0.1.2 * python-automationclient 1.2.1 * rackspace-glanceclient 0.9 * rackspace-novaclient 1.4 If you ignore PyPI and just want to install the base say - 7 services, each one of the python-** clients has its own dependency tree and may or may not build from one of the others. If a vendor wants to extend any of them, it’s basically a fork instead of a clean plugin system. On the CLI front - this would *greatly* simplify the work openstackclient needs to do - it would be able to import from the main SDK and simply provide the noun-verb command line and any other end-user sugar it wanted to. Even if each service wanted to keep its own python-X client instead of relying on
Re: [openstack-dev] a common client library
Hi Doug, Count me in. Climate is currently working on delivering its first python-climateclient but it would be great if we could leverage any olso lib for this. -Sylvain 2014/1/15 Doug Hellmann doug.hellm...@dreamhost.com Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Wed, Jan 15, 2014 at 1:37 PM, Doug Hellmann doug.hellm...@dreamhost.comwrote: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. I would love to see a bit more detail on the structure of the lib(s), the blueprint really doesn't discuss the design/organization/intended API of the libs. For example, I would hope the distinction between the various layers of a client stack don't get lost, i.e. not mixing the low-level REST API bits with the higher-level CLI parsers and decorators. Does the long-term goals include a common caching layer? dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
Great idea, fully support it. We’re interested in that too. One specific thing that was mentioned is the ability to mock auth service seems to be very useful for some test scenarios, we came across that recently. Renat Akhmerov @ Mirantis Inc. On 15 Jan 2014, at 14:07, Sylvain Bauza sylvain.ba...@gmail.com wrote: Hi Doug, Count me in. Climate is currently working on delivering its first python-climateclient but it would be great if we could leverage any olso lib for this. -Sylvain 2014/1/15 Doug Hellmann doug.hellm...@dreamhost.com Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 15, 2014, at 4:55 PM, Renat Akhmerov rakhme...@mirantis.commailto:rakhme...@mirantis.com wrote: Great idea, fully support it. We’re interested in that too. One specific thing that was mentioned is the ability to mock auth service seems to be very useful for some test scenarios, we came across that recently. Renat Akhmerov @ Mirantis Inc. I'd add that verified fakes/mocks/stubs for all network traffic calls are critical for end users wanting to build/test code against the common code/sdk so they don't need a fully running OpenStack install that incurs charges. For fast unit test it's critical On 15 Jan 2014, at 14:07, Sylvain Bauza sylvain.ba...@gmail.commailto:sylvain.ba...@gmail.com wrote: Hi Doug, Count me in. Climate is currently working on delivering its first python-climateclient but it would be great if we could leverage any olso lib for this. -Sylvain 2014/1/15 Doug Hellmann doug.hellm...@dreamhost.commailto:doug.hellm...@dreamhost.com Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Thanks, Doug [1] https://blueprints.launchpad.net/oslo/+spec/common-client-library-2 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.orgmailto:OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Wed, Jan 15, 2014 at 5:39 PM, Dean Troyer dtro...@gmail.com wrote: On Wed, Jan 15, 2014 at 1:37 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. I would love to see a bit more detail on the structure of the lib(s), the blueprint really doesn't discuss the design/organization/intended API of the libs. For example, I would hope the distinction between the various layers of a client stack don't get lost, i.e. not mixing the low-level REST API bits with the higher-level CLI parsers and decorators. Does the long-term goals include a common caching layer? Good questions. I believe the current effort referenced by the blueprint is focusing on moving existing code into the incubator for reuse, to make it easier to restructure later. Alexei, do I have that correct? I was starting to feel like an (unnecessary) bottleneck, so my goal for this email thread was to step out of the middle of the conversation, and make sure everyone who is doing the work knows about each other so you can all coordinate directly. If we need more details for requirements and a design, let's start writing them up. Maybe a wiki page or etherpad to get started, and turn that into blueprints when ready? Who wants to take that on? Doug dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
Doug Hellmann doug.hellm...@dreamhost.com writes: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Solum is already using it https://review.openstack.org/#/c/58067/ I would love to watch this space. Regards, Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
I did notice, however, that neutronclient is conspicuously absent from the Work Items in the blueprint's Whiteboard. It will surely be added later. We already working on several things in parallel and we will add neutronclient soon. I would love to see a bit more detail on the structure of the lib(s), the blueprint really doesn't discuss the design/organization/intended API of the libs. For example, I would hope the distinction between the various layers of a client stack don't get lost, i.e. not mixing the low-level REST API bits with the higher-level CLI parsers and decorators. Does the long-term goals include a common caching layer? Distinction between client layers won't get lost and would only be improved. My basic idea is the following: 1) Transport layer would handle all transport related stuff - HTTP, JSON encoding, auth, caching, etc. 2) Model layer (Resource classes, BaseManager, etc.) will handle data representation, validation 3) API layer will handle all project specific stuff - url mapping, etc. (This will be imported to use client in other applications) 4) Cli level will handle all stuff related to cli mapping - argparse, argcomplete, etc. I believe the current effort referenced by the blueprint is focusing on moving existing code into the incubator for reuse, to make it easier to restructure later. Alexei, do I have that correct? You are right. The first thing we do is try to make all clients look/work in similar way. After we'll continue our work with improving overall structure. 2014/1/16 Noorul Islam K M noo...@noorul.com Doug Hellmann doug.hellm...@dreamhost.com writes: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. If you're interested in this effort, please subscribe to the blueprint and use that to coordinate efforts so we don't produce more than one common library. ;-) Solum is already using it https://review.openstack.org/#/c/58067/ I would love to watch this space. Regards, Noorul ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] a common client library
On Jan 15, 2014, at 6:16 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Wed, Jan 15, 2014 at 5:39 PM, Dean Troyer dtro...@gmail.com wrote: On Wed, Jan 15, 2014 at 1:37 PM, Doug Hellmann doug.hellm...@dreamhost.com wrote: Several people have mentioned to me that they are interested in, or actively working on, code related to a common client library -- something meant to be reused directly as a basis for creating a common library for all of the openstack clients to use. There's a blueprint [1] in oslo, and I believe the keystone devs and unified CLI teams are probably interested in ensuring that the resulting API ends up meeting all of our various requirements. I would love to see a bit more detail on the structure of the lib(s), the blueprint really doesn't discuss the design/organization/intended API of the libs. For example, I would hope the distinction between the various layers of a client stack don't get lost, i.e. not mixing the low-level REST API bits with the higher-level CLI parsers and decorators. Does the long-term goals include a common caching layer? Good questions. I believe the current effort referenced by the blueprint is focusing on moving existing code into the incubator for reuse, to make it easier to restructure later. Alexei, do I have that correct? I was starting to feel like an (unnecessary) bottleneck, so my goal for this email thread was to step out of the middle of the conversation, and make sure everyone who is doing the work knows about each other so you can all coordinate directly. If we need more details for requirements and a design, let's start writing them up. Maybe a wiki page or etherpad to get started, and turn that into blueprints when ready? Who wants to take that on? I’ll take that on: If I schedule an IRC meeting next week; who would like to join? Doug dt -- Dean Troyer dtro...@gmail.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
