Re: [openstack-dev] [k8s][octavia][lbaas] Experiences on using the LB APIs with K8s
> On Mar 16, 2018, at 7:40 AM, Simon Leinen wrote: > > Joe Topjian writes: >> Terraform hat! I want to slightly nit-pick this one since the words >> "leak" and "admin-priv" can sound scary: Terraform technically wasn't >> doing anything wrong. The problem was that Octavia was creating >> resources but not setting ownership to the tenant. When it came time >> to delete the resources, Octavia was correctly refusing, though it >> incorrectly created said resources. > > I dunno... if Octavia created those lower-layer resources on behalf of > the user, then Octavia shouldn't refuse to remove those resources when > the same user later asks it to - independent of what ownership Octavia > chose to apply to those resources. (It would be different it Neutron or > Nova were asked by the user directly to remove the resources created by > Octavia.) > >> From reviewing the discussion, other parties were discovering this >> issue and patching in parallel to your discovery. Both xgerman and >> Vexxhost jumped in to confirm the behavior seen by Terraform. Vexxhost >> quickly applied the patch. It was a really awesome collaboration >> between yourself, dims, xgerman, and Vexxhost. > > Speaking as another operator: Does anyone seriously expect us to deploy > a service (Octavia) in production at a stage where it exhibits this kind > of behavior? Having to clean up leftover resources because the users who > created them cannot remove them is not my idea of fun. (And note that > like most operators, we're a few releases behind, so we might not even > get access to backports IF this gets fixed.) Simon and Joe, one thing that I was not clear on (again, goes back to the statement that mistakes I make are my own), is that this is behavior, admin-scoped resources being created then not released, was seen in the Neutron LBaaSv2 service. The fix _was_ to deploy Octavia and not use the Neutron API. As such, I'm reluctant to use Terraform (or really, any other SDK) to deploy load balancers against the Neutron API. I don't want to be leaking a bunch of resources I can't delete. It's not good for the apps I’m trying to run and it’s definitely not good for the cloud provider. I have much more confidence developing against the Octavia service. We figured this out as a group effort between Vexxhost, Joe, and the Octavia team, and I'm exceptionally grateful to all of them for helping me to sort those issues out. Now, I ultimately dropped it in my own code because I can't rely on the existence of Octavia across all clouds. It had nothing to do with the either the reliability of the GopherCloud/Terraform SDKs or Octavia itself. So, to repeat, leaking admin-scoped resources is a Neutron LBaaSv2 bug, not an Octavia bug. > In our case we're not a compute-oriented cloud provider, and some of our > customers would really like to have a good LBaaS as part of our IaaS > offering. But our experience with this was so-so in the past - for > example, we had to help customers migrate from LBaaSv1 to LBaaSv2. Our > resources (people, tolerance to user-affecting bugs and forced upgrades > etc.) are limited, so we've become careful. > > For users who want to use Kubernetes on our OpenStack service, we rather > point them to Kubernetes's Ingress controller, which performs the LB > function without requiring much from the underlying cloud. Seems like a > fine solution. > -- > Simon. > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [k8s][octavia][lbaas] Experiences on using the LB APIs with K8s
What about the other way around? An Octavia plugin that simply manages k8s Ingress objects on a k8s cluster? Depending on how operators are deploying openstack, this might be a much easier way to deploy Octavia. Thanks, Kevin From: Lingxian Kong [anlin.k...@gmail.com] Sent: Friday, March 16, 2018 5:21 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [k8s][octavia][lbaas] Experiences on using the LB APIs with K8s Just FYI, l7 policy/rule support for Neutron LBaaS V2 and Octavia is on its way[1], because we will have both octavia and magnum deployed on our openstack based public cloud this year, an ingress controller for openstack(octavia) is also on our TODO list, any kind of collaboration are welcomed :-) [1]: https://github.com/gophercloud/gophercloud/pull/833 Cheers, Lingxian Kong (Larry) On Fri, Mar 16, 2018 at 5:01 PM, Joe Topjian mailto:j...@topjian.net>> wrote: Hi Chris, I wear a number of hats related to this discussion, so I'll add a few points of view :) It turns out that with Terraform, it's possible to tear down resources in a way that causes Neutron to leak administrator-privileged resources that can not be deleted by a non-privileged users. In discussions with the Neutron and Octavia teams, it was strongly recommended that I move away from the Neutron LBaaSv2 API and instead adopt Octavia. Vexxhost graciously installed Octavia and my request and I was able to move past this issue. Terraform hat! I want to slightly nit-pick this one since the words "leak" and "admin-priv" can sound scary: Terraform technically wasn't doing anything wrong. The problem was that Octavia was creating resources but not setting ownership to the tenant. When it came time to delete the resources, Octavia was correctly refusing, though it incorrectly created said resources. >From reviewing the discussion, other parties were discovering this issue and >patching in parallel to your discovery. Both xgerman and Vexxhost jumped in to >confirm the behavior seen by Terraform. Vexxhost quickly applied the patch. It >was a really awesome collaboration between yourself, dims, xgerman, and >Vexxhost. This highlights the first call to action for our public and private cloud community: encouraging the rapid migration from older, unsupported APIs to Octavia. Operator hat! The clouds my team and I run are more compute-based. Our users would be more excited if we increased our GPU pool than enhanced the networking services. With that in mind, when I hear it said that "Octavia is backwards-compatible with Neutron LBaaS v2", I think "well, cool, that means we can keep running Neutron LBaaS v2 for now" and focus our efforts elsewhere. I totally get why Octavia is advertised this way and it's very much appreciated. When I learned about Octavia, my knee-jerk reaction was "oh no, not another load balancer" but that was remedied when I learned it's more like LBaaSv2++. I'm sure we'll deploy Octavia some day, but it's not our primary focus and we can still squeak by with Neutron's LBaaS v2. If you *really* wanted us to deploy Octavia ASAP, then a migration guide would be wonderful. I read over the "Developer / Operator Quick Start Guide" and found it very well written! I groaned over having to build an image but I also really appreciate the image builder script. If there can't be pre-built images available for testing, the second-best option is that script. This highlights a second call to action for the SDK and provider developers: recognizing the end of life of the Neutron LBaaSv2 API[4][5] and adding support for more advanced Octavia features. Gophercloud hat! We've supported Octavia for a few months now, but purely by having the load-balancer client piggyback off of the Neutron LBaaS v2 API. We made the decision this morning, coincidentally enough, to have Octavia be a first-class service peered with Neutron rather than think of Octavia as a Neutron/network child. This will allow Octavia to fully flourish without worry of affecting the existing LBaaS v2 API (which we'll still keep around separately). Thanks, Joe __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe<http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [k8s][octavia][lbaas] Experiences on using the LB APIs with K8s
Joe Topjian writes: > Terraform hat! I want to slightly nit-pick this one since the words > "leak" and "admin-priv" can sound scary: Terraform technically wasn't > doing anything wrong. The problem was that Octavia was creating > resources but not setting ownership to the tenant. When it came time > to delete the resources, Octavia was correctly refusing, though it > incorrectly created said resources. I dunno... if Octavia created those lower-layer resources on behalf of the user, then Octavia shouldn't refuse to remove those resources when the same user later asks it to - independent of what ownership Octavia chose to apply to those resources. (It would be different it Neutron or Nova were asked by the user directly to remove the resources created by Octavia.) > From reviewing the discussion, other parties were discovering this > issue and patching in parallel to your discovery. Both xgerman and > Vexxhost jumped in to confirm the behavior seen by Terraform. Vexxhost > quickly applied the patch. It was a really awesome collaboration > between yourself, dims, xgerman, and Vexxhost. Speaking as another operator: Does anyone seriously expect us to deploy a service (Octavia) in production at a stage where it exhibits this kind of behavior? Having to clean up leftover resources because the users who created them cannot remove them is not my idea of fun. (And note that like most operators, we're a few releases behind, so we might not even get access to backports IF this gets fixed.) In our case we're not a compute-oriented cloud provider, and some of our customers would really like to have a good LBaaS as part of our IaaS offering. But our experience with this was so-so in the past - for example, we had to help customers migrate from LBaaSv1 to LBaaSv2. Our resources (people, tolerance to user-affecting bugs and forced upgrades etc.) are limited, so we've become careful. For users who want to use Kubernetes on our OpenStack service, we rather point them to Kubernetes's Ingress controller, which performs the LB function without requiring much from the underlying cloud. Seems like a fine solution. -- Simon. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [k8s][octavia][lbaas] Experiences on using the LB APIs with K8s
On Fri, Mar 16, 2018 at 5:01 AM, Joe Topjian wrote: > Hi Chris, > > I wear a number of hats related to this discussion, so I'll add a few > points of view :) > > It turns out that with >> Terraform, it's possible to tear down resources in a way that causes >> Neutron to >> leak administrator-privileged resources that can not be deleted by a >> non-privileged users. In discussions with the Neutron and Octavia teams, >> it was >> strongly recommended that I move away from the Neutron LBaaSv2 API and >> instead >> adopt Octavia. Vexxhost graciously installed Octavia and my request and I >> was >> able to move past this issue. >> > > Terraform hat! I want to slightly nit-pick this one since the words "leak" > and "admin-priv" can sound scary: Terraform technically wasn't doing > anything wrong. The problem was that Octavia was creating resources but not > setting ownership to the tenant. When it came time to delete the resources, > Octavia was correctly refusing, though it incorrectly created said > resources. > > From reviewing the discussion, other parties were discovering this issue > and patching in parallel to your discovery. Both xgerman and Vexxhost > jumped in to confirm the behavior seen by Terraform. Vexxhost quickly > applied the patch. It was a really awesome collaboration between yourself, > dims, xgerman, and Vexxhost. > > >> This highlights the first call to action for our public and private cloud >> community: encouraging the rapid migration from older, unsupported APIs to >> Octavia. >> > > Operator hat! The clouds my team and I run are more compute-based. Our > users would be more excited if we increased our GPU pool than enhanced the > networking services. With that in mind, when I hear it said that "Octavia > is backwards-compatible with Neutron LBaaS v2", I think "well, cool, that > means we can keep running Neutron LBaaS v2 for now" and focus our efforts > elsewhere. > > I totally get why Octavia is advertised this way and it's very much > appreciated. When I learned about Octavia, my knee-jerk reaction was "oh > no, not another load balancer" but that was remedied when I learned it's > more like LBaaSv2++. I'm sure we'll deploy Octavia some day, but it's not > our primary focus and we can still squeak by with Neutron's LBaaS v2. > > If you *really* wanted us to deploy Octavia ASAP, then a migration guide > would be wonderful. I read over the "Developer / Operator Quick Start > Guide" and found it very well written! I groaned over having to build an > image but I also really appreciate the image builder script. If there can't > be pre-built images available for testing, the second-best option is that > script. > Periodic builds of Ubuntu and CentOS pre-built test images coming soon: https://review.openstack.org/#/c/549259/ Periodic builds by the RDO project: https://images.rdoproject.org/octavia/master/ ( https://review.rdoproject.org/r/#/c/11805/) > >> This highlights a second call to action for the SDK and provider >> developers: >> recognizing the end of life of the Neutron LBaaSv2 API[4][5] and adding >> support for more advanced Octavia features. >> > > Gophercloud hat! We've supported Octavia for a few months now, but purely > by having the load-balancer client piggyback off of the Neutron LBaaS v2 > API. We made the decision this morning, coincidentally enough, to have > Octavia be a first-class service peered with Neutron rather than think of > Octavia as a Neutron/network child. This will allow Octavia to fully > flourish without worry of affecting the existing LBaaS v2 API (which we'll > still keep around separately). > > Thanks, > Joe > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [k8s][octavia][lbaas] Experiences on using the LB APIs with K8s
Just FYI, l7 policy/rule support for Neutron LBaaS V2 and Octavia is on its way[1], because we will have both octavia and magnum deployed on our openstack based public cloud this year, an ingress controller for openstack(octavia) is also on our TODO list, any kind of collaboration are welcomed :-) [1]: https://github.com/gophercloud/gophercloud/pull/833 Cheers, Lingxian Kong (Larry) On Fri, Mar 16, 2018 at 5:01 PM, Joe Topjian wrote: > Hi Chris, > > I wear a number of hats related to this discussion, so I'll add a few > points of view :) > > It turns out that with >> Terraform, it's possible to tear down resources in a way that causes >> Neutron to >> leak administrator-privileged resources that can not be deleted by a >> non-privileged users. In discussions with the Neutron and Octavia teams, >> it was >> strongly recommended that I move away from the Neutron LBaaSv2 API and >> instead >> adopt Octavia. Vexxhost graciously installed Octavia and my request and I >> was >> able to move past this issue. >> > > Terraform hat! I want to slightly nit-pick this one since the words "leak" > and "admin-priv" can sound scary: Terraform technically wasn't doing > anything wrong. The problem was that Octavia was creating resources but not > setting ownership to the tenant. When it came time to delete the resources, > Octavia was correctly refusing, though it incorrectly created said > resources. > > From reviewing the discussion, other parties were discovering this issue > and patching in parallel to your discovery. Both xgerman and Vexxhost > jumped in to confirm the behavior seen by Terraform. Vexxhost quickly > applied the patch. It was a really awesome collaboration between yourself, > dims, xgerman, and Vexxhost. > > >> This highlights the first call to action for our public and private cloud >> community: encouraging the rapid migration from older, unsupported APIs to >> Octavia. >> > > Operator hat! The clouds my team and I run are more compute-based. Our > users would be more excited if we increased our GPU pool than enhanced the > networking services. With that in mind, when I hear it said that "Octavia > is backwards-compatible with Neutron LBaaS v2", I think "well, cool, that > means we can keep running Neutron LBaaS v2 for now" and focus our efforts > elsewhere. > > I totally get why Octavia is advertised this way and it's very much > appreciated. When I learned about Octavia, my knee-jerk reaction was "oh > no, not another load balancer" but that was remedied when I learned it's > more like LBaaSv2++. I'm sure we'll deploy Octavia some day, but it's not > our primary focus and we can still squeak by with Neutron's LBaaS v2. > > If you *really* wanted us to deploy Octavia ASAP, then a migration guide > would be wonderful. I read over the "Developer / Operator Quick Start > Guide" and found it very well written! I groaned over having to build an > image but I also really appreciate the image builder script. If there can't > be pre-built images available for testing, the second-best option is that > script. > > >> This highlights a second call to action for the SDK and provider >> developers: >> recognizing the end of life of the Neutron LBaaSv2 API[4][5] and adding >> support for more advanced Octavia features. >> > > Gophercloud hat! We've supported Octavia for a few months now, but purely > by having the load-balancer client piggyback off of the Neutron LBaaS v2 > API. We made the decision this morning, coincidentally enough, to have > Octavia be a first-class service peered with Neutron rather than think of > Octavia as a Neutron/network child. This will allow Octavia to fully > flourish without worry of affecting the existing LBaaS v2 API (which we'll > still keep around separately). > > Thanks, > Joe > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [k8s][octavia][lbaas] Experiences on using the LB APIs with K8s
Hi Chris, I wear a number of hats related to this discussion, so I'll add a few points of view :) It turns out that with > Terraform, it's possible to tear down resources in a way that causes > Neutron to > leak administrator-privileged resources that can not be deleted by a > non-privileged users. In discussions with the Neutron and Octavia teams, > it was > strongly recommended that I move away from the Neutron LBaaSv2 API and > instead > adopt Octavia. Vexxhost graciously installed Octavia and my request and I > was > able to move past this issue. > Terraform hat! I want to slightly nit-pick this one since the words "leak" and "admin-priv" can sound scary: Terraform technically wasn't doing anything wrong. The problem was that Octavia was creating resources but not setting ownership to the tenant. When it came time to delete the resources, Octavia was correctly refusing, though it incorrectly created said resources. >From reviewing the discussion, other parties were discovering this issue and patching in parallel to your discovery. Both xgerman and Vexxhost jumped in to confirm the behavior seen by Terraform. Vexxhost quickly applied the patch. It was a really awesome collaboration between yourself, dims, xgerman, and Vexxhost. > This highlights the first call to action for our public and private cloud > community: encouraging the rapid migration from older, unsupported APIs to > Octavia. > Operator hat! The clouds my team and I run are more compute-based. Our users would be more excited if we increased our GPU pool than enhanced the networking services. With that in mind, when I hear it said that "Octavia is backwards-compatible with Neutron LBaaS v2", I think "well, cool, that means we can keep running Neutron LBaaS v2 for now" and focus our efforts elsewhere. I totally get why Octavia is advertised this way and it's very much appreciated. When I learned about Octavia, my knee-jerk reaction was "oh no, not another load balancer" but that was remedied when I learned it's more like LBaaSv2++. I'm sure we'll deploy Octavia some day, but it's not our primary focus and we can still squeak by with Neutron's LBaaS v2. If you *really* wanted us to deploy Octavia ASAP, then a migration guide would be wonderful. I read over the "Developer / Operator Quick Start Guide" and found it very well written! I groaned over having to build an image but I also really appreciate the image builder script. If there can't be pre-built images available for testing, the second-best option is that script. > This highlights a second call to action for the SDK and provider > developers: > recognizing the end of life of the Neutron LBaaSv2 API[4][5] and adding > support for more advanced Octavia features. > Gophercloud hat! We've supported Octavia for a few months now, but purely by having the load-balancer client piggyback off of the Neutron LBaaS v2 API. We made the decision this morning, coincidentally enough, to have Octavia be a first-class service peered with Neutron rather than think of Octavia as a Neutron/network child. This will allow Octavia to fully flourish without worry of affecting the existing LBaaS v2 API (which we'll still keep around separately). Thanks, Joe __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [k8s][octavia][lbaas] Experiences on using the LB APIs with K8s
As I've been working more in the Kubernetes community, I've been evaluating the different points of integration between OpenStack services and the Kubernetes application platform. One of the weaker points of integration has been in using the OpenStack LBaaS APIs to create load balancers for Kubernetes applications. Using this as a framing device, I'd like to begin a discussion about the general development, deployment, and usage of the LBaaS API and how different parts of our community can rally around and strengthen the API in the coming year. I'd like to note right from the beginning that this isn't a disparagement of the fantastic work that's being done by the Octavia team, but rather an evaluation of the current state of the API and a call to our rich community of developers, cloud deployers, users, and app developers to help move the API to a place where it is expected to be present and shows the same level of consistency across deployments that we see with the Nova, Cinder, and Neutron core APIs. The seed of this discussion comes from my efforts to enable third-party Kubernetes cloud provider testing, as well as discussions with the Kubernetes-SIG-OpenStack community in the #sig-openstack Slack channel in the Kubernetes organization[0]. As a full disclaimer, my recounting of this discussion represents my own impressions, and although I mention active participants by name I do not represent their views. Any mistakes I make are my own. To set the stage, Kubernetes uses a third-party load-balancer service (either from a Kubernetes hosted application or from a cloud-provider API) to provide high-availability for the applications it manages. The OpenStack provider offers a generic interface to the LBaaSv2, with an option to enable Octavia instead of the Neutron API. The provider is build off of the GopherCloud SDK. In my own efforts to enable testing of this provider, I'm using Terraform to orchestrate the K8s deployment and installation. Since I needed to use a public cloud provider to turn this automated testing over to a third party, I chose Vexxhost, as they have been generous donors in this effort for the the CloudLab efforts in general, and have provided tremendous support in debugging problems I've run in to. The first major issue I ran in to was a race condition in using the Neutron LBaaSv2 API. It turns out that with Terraform, it's possible to tear down resources in a way that causes Neutron to leak administrator-privileged resources that can not be deleted by a non-privileged users. In discussions with the Neutron and Octavia teams, it was strongly recommended that I move away from the Neutron LBaaSv2 API and instead adopt Octavia. Vexxhost graciously installed Octavia and my request and I was able to move past this issue. This raises a fundamental issue facing our community with regards to the load balancer APIs: there is little consistency as to which API is deployed, and we have installations that still deploy on the LBaaSv1 API. Indeed, the OpenStack User Survey reported in November of 2017 that only 7% of production installations were running Octavia[1]. Meanwhile, Neutron LBaaSv1 was deprecated in Liberty, and Neutron LBaaSv2 was recently deprecated in the Queens release. The lack of a migration path from v1 to v2 helped to slow adoption, and the additional requirements for installing Octavia has also been a factor in increasing adoption of the supported LBaaSv2 implementation. This highlights the first call to action for our public and private cloud community: encouraging the rapid migration from older, unsupported APIs to Octavia. Because of this wide range of deployed APIs, I changed my own deployment code to launch a user-space VM and install a non-tls-terminating Nginx load balancer for my Kubernetes control plane[2]. I'm not the only person who has adopted an approach like this. In the #sig-openstack channel, Saverio Proto (zioproto) discussed how he uses the K8s Nginx ingress load balancer[3] in favor of the OpenStack provider load balancer. My take away from his description is that it's preferable to use the K8s-based ingress load balancer because: * The common LBaaSv2 API does not support TLS termination. * You don't need provision an additional virtual machine. * You aren't dependent on an appropriate and supported API being available on your cloud. German Eichberger (xgerman) and Adam Harwell (rm_you) from the Octavia team were present for the discussion, and presented a strong case for using the Octavia APIs. My take away was: * Octavia does support TLS termination, and it's the dependence on the Neutron API that removes the ability to take advantage of it. * It provides a lot more than just a "VM with haproxy", and has stability guarantees. This highlights a second call to action for the SDK and provider developers: recognizing the end of life of the Neutron LBaaSv2 API[4][5] and adding support for more advanced Octavia features. As part of
