Re: [openstack-dev] Pecan Evaluation for Marconi
Kurt Griffiths wrote: Kudos to Balaji for working so hard on this. I really appreciate his candid feedback on both frameworks. Indeed, that analysis is very much appreciated. From the Technical Committee perspective, we put a high weight on a factor that was not included in the report results: consistency and convergence between projects we commonly release in an integrated manner every 6 months. There was historically a lot of deviation, but as we add more projects that deviation is becoming more costly. We want developers to be able to jump from one project to another easily, and we want convergence from an operators perspective. Individual projects are obviously allowed to pick the best tool in their toolbox. But the TC may also decide to let projects live out of the integrated release if we feel they would add too much divergence in. After reviewing the report below, I would recommend that Marconi continue using Falcon for the v1.1 API and then re-evaluate Pecan for v2.0 or possibly look at using swob. The report (and your email below) makes a compelling argument that Falcon is a better match for Marconi's needs (or for a data-plane API) than Pecan currently is. My question would be, can Pecan be improved to also cover Marconi's use case ? Could we have the best of both worlds (an appropriate tool *and* convergence) ? If the answer is yes, probably, then it might be an option to delay inclusion in the integrated release so that we don't add (even temporary) divergence. If the answer is definitely no, then we'll have to choose between convergence and functionality. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
On 19/03/14 12:31 +0100, Thierry Carrez wrote: Kurt Griffiths wrote: Kudos to Balaji for working so hard on this. I really appreciate his candid feedback on both frameworks. Indeed, that analysis is very much appreciated. From the Technical Committee perspective, we put a high weight on a factor that was not included in the report results: consistency and convergence between projects we commonly release in an integrated manner every 6 months. There was historically a lot of deviation, but as we add more projects that deviation is becoming more costly. We want developers to be able to jump from one project to another easily, and we want convergence from an operators perspective. Individual projects are obviously allowed to pick the best tool in their toolbox. But the TC may also decide to let projects live out of the integrated release if we feel they would add too much divergence in. My only concern in this case - I'm not sure if this has been discussed or written somewhere - is to define what the boundaries of that divergence are. For instance, and I know this will sound quite biased, I don't think there's anything wrong on supporting a *set* of wsgi frameworks. To be fair, there's already a set since currently integrated projects use webob, swob and Pecan. The point I'd like to get at is that as a general rule we probably shouldn't limit the set of supported libraries to just 1. Cheers, Flavio -- @flaper87 Flavio Percoco pgp6sx_pc63_r.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
Thierry Carrez wrote: There was historically a lot of deviation, but as we add more projects that deviation is becoming more costly. I totally understand the benefits of reducing the variance between projects, and to be sure, I am not suggesting we have 10 different libraries to do X. However, as more projects are added, the variety of requirements also increases, and it becomes very difficult for a single library to meet all the projects' needs without some projects having to make non-trivial compromises. One approach to this that I’ve seen work well in other communities is to define a small set of options that cover the major use cases. My question would be, can Pecan be improved to also cover Marconi's use case ? Could we have the best of both worlds (an appropriate tool *and* convergence) ? That would certainly be ideal, but as always, the devil is in the details. Pecan performance has been improving, so on that front there may be an opportunity for convergence (assuming webob also improves in performance). However, with respect to code paths and dependencies, I am not clear on the path forward. Some dependencies could be removed by creating some kind of “pecan-light” library, but that would need to be done in a way that does not break projects that rely on those extra features. That would still leave webob, which is an often-used selling point for Pecan. I am not confident that webob can be modified to address Marconi and Swift's needs without making backwards-incompatible changes to the library which would obviously not be acceptable to the broader Python community. Kurt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
On Mar 19, 2014, at 10:18 AM, Kurt Griffiths kurt.griffi...@rackspace.com wrote: Thierry Carrez wrote: There was historically a lot of deviation, but as we add more projects that deviation is becoming more costly. I totally understand the benefits of reducing the variance between projects, and to be sure, I am not suggesting we have 10 different libraries to do X. However, as more projects are added, the variety of requirements also increases, and it becomes very difficult for a single library to meet all the projects' needs without some projects having to make non-trivial compromises. One approach to this that I’ve seen work well in other communities is to define a small set of options that cover the major use cases. My question would be, can Pecan be improved to also cover Marconi's use case ? Could we have the best of both worlds (an appropriate tool *and* convergence) ? That would certainly be ideal, but as always, the devil is in the details. Pecan performance has been improving, so on that front there may be an opportunity for convergence (assuming webob also improves in performance). However, with respect to code paths and dependencies, I am not clear on the path forward. Some dependencies could be removed by creating some kind of “pecan-light” library, but that would need to be done in a way that does not break projects that rely on those extra features. That would still leave webob, which is an often-used selling point for Pecan. I am not confident that webob can be modified to address Marconi and Swift's needs without making backwards-incompatible changes to the library which would obviously not be acceptable to the broader Python community. I’m not sure that “number of dependencies” is a useful metric at all tbh. At the very least it’s not a very telling metric in the way it was presented in the review. An example - A tool that has to safely render untrusted HTML, you could do it with nothing but the stdlib using say regex based parsers (and get it wrong) or you could depend on bleach which depends on html5lib. Using the “number of dependencies” metric the first would be considered the superior method however it’s deeply flawed. The reason given in the report is that more dependencies = larger attack surface, but that’s not really accurate either. Often times you’ll find that if two libraries solve the same problems, one with dependencies and one without the one without dependencies has a less battle tested reimplementation of whatever functionality the other library has a dependency for. In order to accurately assess the impact of dependencies you have to understand what the library is using those dependencies for, how well tested those dependencies are, what the release cycle and backwards compatibility policy of those dependencies are, and what the other project is doing in place of a dependency for the feature(s) that depend on them (which the answer may be that it doesn’t have that feature, and then you have to decide if that feature is useful to you and if you’ll need to add a dependency or write less battle tested code in order to get it). - Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
On Wed, Mar 19, 2014 at 7:31 AM, Thierry Carrez thie...@openstack.orgwrote: Kurt Griffiths wrote: Kudos to Balaji for working so hard on this. I really appreciate his candid feedback on both frameworks. Indeed, that analysis is very much appreciated. From the Technical Committee perspective, we put a high weight on a factor that was not included in the report results: consistency and convergence between projects we commonly release in an integrated manner every 6 months. There was historically a lot of deviation, but as we add more projects that deviation is becoming more costly. We want developers to be able to jump from one project to another easily, and we want convergence from an operators perspective. Individual projects are obviously allowed to pick the best tool in their toolbox. But the TC may also decide to let projects live out of the integrated release if we feel they would add too much divergence in. As Thierry points out, an important aspect of being in the integrated release is being aligned with the rest of the community. The evaluation gives community considerations the lowest weight among the criteria considered. Does that ranking reflect the opinion of the entire Marconi team? If so, what benefits do you see to being integrated? The evaluation does not discuss any of the infrastructure tooling being built up around OpenStack's use of Pecan. For example, what will Marconi do for API documentation generation? Pecan is currently gating changes against projects that use it, so we can be sure that changes to the framework do not break our applications. This does not appear to have been factored into the evaluation. After reviewing the report below, I would recommend that Marconi continue using Falcon for the v1.1 API and then re-evaluate Pecan for v2.0 or possibly look at using swob. The report (and your email below) makes a compelling argument that Falcon is a better match for Marconi's needs (or for a data-plane API) than Pecan currently is. My question would be, can Pecan be improved to also cover Marconi's use case ? Could we have the best of both worlds (an appropriate tool *and* convergence) ? We had several conversations with Kurt and Flavio in Hong Kong about adding features to Pecan to support the Marconi team, and Ryan prototyped some of those changes shortly after we returned home. Was any of that work considered in the evaluation? Doug If the answer is yes, probably, then it might be an option to delay inclusion in the integrated release so that we don't add (even temporary) divergence. If the answer is definitely no, then we'll have to choose between convergence and functionality. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
On Wed, Mar 19, 2014 at 10:11 AM, Flavio Percoco fla...@redhat.com wrote: On 19/03/14 12:31 +0100, Thierry Carrez wrote: Kurt Griffiths wrote: Kudos to Balaji for working so hard on this. I really appreciate his candid feedback on both frameworks. Indeed, that analysis is very much appreciated. From the Technical Committee perspective, we put a high weight on a factor that was not included in the report results: consistency and convergence between projects we commonly release in an integrated manner every 6 months. There was historically a lot of deviation, but as we add more projects that deviation is becoming more costly. We want developers to be able to jump from one project to another easily, and we want convergence from an operators perspective. Individual projects are obviously allowed to pick the best tool in their toolbox. But the TC may also decide to let projects live out of the integrated release if we feel they would add too much divergence in. My only concern in this case - I'm not sure if this has been discussed or written somewhere - is to define what the boundaries of that divergence are. For instance, and I know this will sound quite biased, I don't think there's anything wrong on supporting a *set* of wsgi frameworks. To be fair, there's already a set since currently integrated projects use webob, swob and Pecan. Only one project is using swob, and it is unlikely that will change. The other projects are mostly using the legacy oslo framework or Pecan, although a few are using Flask (perhaps based on ceilometer's initial experimentation with it?). As I understand it, all of the integrated projects have looked at Pecan, and are anticipating the transition. Most have no reason to create a new API version, and therefore build a new API service to avoid introducing incompatibilities by rebuilding the existing API with a new tool. This aligns with the plan when Pecan was proposed as a standard. Doug The point I'd like to get at is that as a general rule we probably shouldn't limit the set of supported libraries to just 1. Cheers, Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
On 19/03/14 11:20 -0400, Doug Hellmann wrote: On Wed, Mar 19, 2014 at 10:11 AM, Flavio Percoco fla...@redhat.com wrote: My only concern in this case - I'm not sure if this has been discussed or written somewhere - is to define what the boundaries of that divergence are. For instance, and I know this will sound quite biased, I don't think there's anything wrong on supporting a *set* of wsgi frameworks. To be fair, there's already a set since currently integrated projects use webob, swob and Pecan. Only one project is using swob, and it is unlikely that will change. The other projects are mostly using the legacy oslo framework or Pecan, although a few are using Flask (perhaps based on ceilometer's initial experimentation with it?). As I understand it, all of the integrated projects have looked at Pecan, and are anticipating the transition. Most have no reason to create a new API version, and therefore build a new API service to avoid introducing incompatibilities by rebuilding the existing API with a new tool. This aligns with the plan when Pecan was proposed as a standard. Yeah, what I wanted to say is that it is arguable that we should add another framework (falcon) to this already existing set of frameworks. Although, it is being reduced to just 2 which still raises my previous question: The point I'd like to get at is that as a general rule we probably shouldn't limit the set of supported libraries to just 1. ... but perhaps decide in a per-case basis. Cheers, Flavio -- @flaper87 Flavio Percoco pgpXcar_3te57.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
On 03/19/2014 08:20 AM, Doug Hellmann wrote: As I understand it, all of the integrated projects have looked at Pecan, and are anticipating the transition. Most have no reason to create a new API version, and therefore build a new API service to avoid introducing incompatibilities by rebuilding the existing API with a new tool. This aligns with the plan when Pecan was proposed as a standard. Doug I have evaluated it for Cinder and have spoke to numerous interested folks in Cinder about using Pecan. It's currently what we're planning to move to after I did a rough prototype for some of our core API controllers. As you mentioned, we have no reason to do a version bump yet. We'll likely do a bump to be py3 compatible rather than for a significant change. -- Mike Perez ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
Only one project is using swob, and it is unlikely that will change. That begs the question, *why* is that unlikely to change? Is it because there are fundamental needs that are not met by Pecan? If I understand the original charter for Oslo, it was to consolidate code already in use by projects to facilitate sharing. It would seem to me that if swob has a compelling reason to exist, and other data plane projects see value in it (and I’m starting to think Marconi would be on that list), it would be a good candidate for extraction to a standalone library. I personally see a lot of alignment between swob and Falcon, and convergence between the two libraries could be a productive path to explore. Kurt ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
On Wed, Mar 19 2014, Donald Stufft wrote: I’m not sure that “number of dependencies” is a useful metric at all tbh. At the very least it’s not a very telling metric in the way it was presented in the review. […] +1000 Seriously, this in itself just discredits any value in this analysis to me. -- Julien Danjou # Free Software hacker # http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
On Wed, Mar 19 2014, Kurt Griffiths wrote: That begs the question, *why* is that unlikely to change? Because that project is Swift. -- Julien Danjou // Free Software hacker // http://julien.danjou.info signature.asc Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
On 03/19/2014 01:47 PM, Mike Perez wrote: On 03/19/2014 08:20 AM, Doug Hellmann wrote: As I understand it, all of the integrated projects have looked at Pecan, and are anticipating the transition. Most have no reason to create a new API version, and therefore build a new API service to avoid introducing incompatibilities by rebuilding the existing API with a new tool. This aligns with the plan when Pecan was proposed as a standard. Doug I have evaluated it for Cinder and have spoke to numerous interested folks in Cinder about using Pecan. It's currently what we're planning to move to after I did a rough prototype for some of our core API controllers. As you mentioned, we have no reason to do a version bump yet. We'll likely do a bump to be py3 compatible rather than for a significant change. FWIW, we've also discussed it for Nova. We approved converting to use it for the v3 API that is still being worked on. I hope to see that get movement again during Juno. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
On Wed, Mar 19, 2014 at 10:00 AM, Doug Hellmann doug.hellm...@dreamhost.com wrote: On Wed, Mar 19, 2014 at 7:31 AM, Thierry Carrez thie...@openstack.orgwrote: Kurt Griffiths wrote: Kudos to Balaji for working so hard on this. I really appreciate his candid feedback on both frameworks. Indeed, that analysis is very much appreciated. From the Technical Committee perspective, we put a high weight on a factor that was not included in the report results: consistency and convergence between projects we commonly release in an integrated manner every 6 months. There was historically a lot of deviation, but as we add more projects that deviation is becoming more costly. We want developers to be able to jump from one project to another easily, and we want convergence from an operators perspective. Individual projects are obviously allowed to pick the best tool in their toolbox. But the TC may also decide to let projects live out of the integrated release if we feel they would add too much divergence in. As Thierry points out, an important aspect of being in the integrated release is being aligned with the rest of the community. The evaluation gives community considerations the lowest weight among the criteria considered. Does that ranking reflect the opinion of the entire Marconi team? If so, what benefits do you see to being integrated? The evaluation does not discuss any of the infrastructure tooling being built up around OpenStack's use of Pecan. For example, what will Marconi do for API documentation generation? Doug and I talked about this in #openstack-dev today, and I just wanted to point out that only one of nine integrated projects uses a Pecan-based solution for API documentation generation, using a tool called sphinxcontrib-docbookrestapi. [1] I consider this question a bit of a false representation of the direction we're going with API docs. There's no standard yet established other than somehow create WADL so we can accurately represent a reference listing to users. Also with extensible APIs it might be easier to just maintain WADL, we just don't know until we get more data from more teams using the Sphinx extension. That said, we do use a common toolset to generate configuration reference information and I'd expect all integrated projects to save time and effort by standardizing as much as possible. The Marconi team has had a tech writer assigned and the team is working within the guidelines we've given them. Thanks, Anne 1. https://github.com/enovance/sphinxcontrib-docbookrestapi I'd like to rename it to sphinxcontrib-restapi or some such, since it doesn't generate docbook. Pecan is currently gating changes against projects that use it, so we can be sure that changes to the framework do not break our applications. This does not appear to have been factored into the evaluation. After reviewing the report below, I would recommend that Marconi continue using Falcon for the v1.1 API and then re-evaluate Pecan for v2.0 or possibly look at using swob. The report (and your email below) makes a compelling argument that Falcon is a better match for Marconi's needs (or for a data-plane API) than Pecan currently is. My question would be, can Pecan be improved to also cover Marconi's use case ? Could we have the best of both worlds (an appropriate tool *and* convergence) ? We had several conversations with Kurt and Flavio in Hong Kong about adding features to Pecan to support the Marconi team, and Ryan prototyped some of those changes shortly after we returned home. Was any of that work considered in the evaluation? Doug If the answer is yes, probably, then it might be an option to delay inclusion in the integrated release so that we don't add (even temporary) divergence. If the answer is definitely no, then we'll have to choose between convergence and functionality. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
On Mar 19, 2014, at 12:27 PM, Julien Danjou jul...@danjou.info wrote: On Wed, Mar 19 2014, Kurt Griffiths wrote: That begs the question, *why* is that unlikely to change? Because that project is Swift. If you look at the Swift code, you'll see that swob is not a replacement for either Pecan or Falcon. swob was written to replace WebOb, and we documented why we did this. https://github.com/openstack/swift/blob/master/swift/common/swob.py#L23 It's an in-tree module written to remove a recurring pain point. swob has allowed the Swift team to focus their time on adding features and fixing bugs in other parts of the code. Why don't we use Pecan or Falcon in Swift? Mostly because we don't need the functionality that they provide, and so there is no reason to go add a dependency (and thus increase packaging and install requirements on deployers). Now if there are other uses for swob outside of Swift, let's have a conversation about including it in an external library so we can all benefit. --- The comparison that Balaji did between Falcon and Pecan looks like a very good overview. It gives information necessary to make an informed choice based on real data instead of it's what everybody is doing. If you don't like some criteria reported on, I'm sure Balaji would be happy to see your comparison and evaluation. We all want to make informed decisions based on data, not claims. Balaji's analysis is a great start on figuring out what the Marconi project should choose. As such, it seems that the Marconi team is the responsible party to make the right choice for their use case, after weighing all the factors. --John signature.asc Description: Message signed with OpenPGP using GPGMail ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Pecan Evaluation for Marconi
After reviewing the report below, I would recommend that Marconi continue using Falcon for the v1.1 API and then re-evaluate Pecan for v2.0 or possibly look at using swob. I wanted to post my recommendation to the general list, because my request to continue using Falcon speaks to a broader issue. I think the community can agree that different projects have different needs. A good craftsman has more than one type of hammer in his toolbox. Most of the OpenStack APIs to date have been control-plane APIs, not data plane. Swift is a notable exception. Falcon was created to address the needs of a high-traffic data-plane API, while Pecan’s history would suggest that it was conceived as a solution for building web apps and control plane APIs. Two major differentiators between these types of APIs: 1. Performance (i.e., latency, throughput, efficiency). With a data plane API, every ms counts, esp. when it comes to running a large cloud. 2. Diagnostics. When your service is piping a huge number of requests/sec, you become very susceptible to edge cases. Also, the amount of downtime your users will tolerate is quite low, since even a small hiccup means a whole lot of work can’t get done. Having a smaller code base, minimizing dependencies, and making the code that is there as straightforward, predictable and debuggable as possible becomes very important in situations like these. Falcon and swob were created to address these needs, hence their use in data plane APIs. I’d love to get everyone’s thoughts on the requirements of data plane APIs they have been involved with (not necessarily OpenStack projects). Kurt From: Balaji Iyer balaji.i...@rackspace.commailto:balaji.i...@rackspace.com Reply-To: OpenStack Dev openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Tuesday, March 18, 2014 at 11:55 AM To: OpenStack Dev openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: [openstack-dev] [Marconi] Pecan Evaluation for Marconi I work for Rackspace and Im fairly new to Openstack Ecosystem. Recently, I came across an opportunity to evaluate Pecan for Marconi and produce a comprehensive report. I have not worked with Pecan or Falcon prior to this evaluation, and have no vested interest in these two frameworks. Evaluating frameworks is not always easy, but I have strived to cover as many details as applicable. I have evaluated Pecan and Falcon only on how it fits Marconi and this should not be treated as a general evaluation for all products. It is always recommended to evaluate frameworks based on your product's requirements and its workload. Benchmarking is not always easy, hence I have spent a good amount of time benchmarking these two frameworks using different tools and under different network and load conditions with Marconi. Some of the experiences I have mentioned in the report are very subjective and it narrates mine - you may have had a different experience with these frameworks, which is totally acceptable. Full evaluation report is available here - https://wiki.openstack.org/wiki/Marconi/pecan-evaluation Thought of sharing this with the community in the hope that someone may find this useful. Thanks, Balaji Iyer ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
