Re: [openstack-dev] [Massively distributed] Optimizing the interaction of OpenStack components
On 5/24/2017 9:59 AM, Matt Riedemann wrote: I started going down a path the other night of trying to see if we could bulk query floating IPs when building the internal instance network info cache [1] but it looks like that's not supported. The REST API docs for Neutron say that you can't OR filter query parameters together, but at the time looking at the code it seemed like it might be possible. Kevin Benton pointed out the bug in my code, so the bulk query for floating IPs is working now it seems: https://review.openstack.org/#/c/465792/ http://logs.openstack.org/92/465792/3/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fd0e93f/logs/screen-q-svc.txt.gz#_May_24_20_54_02_457529 So we can probably iterate on that a bit to bulk query other things, but I'd have to dig through the code to see where we're doing that. -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Massively distributed] Optimizing the interaction of OpenStack components
On 5/11/2017 1:44 PM, Georg Kunz wrote: Nevertheless, one concrete thing which came to my mind, is this proposed improvement of the interaction between Nova and Neutron: https://review.openstack.org/#/c/390513/ In a nutshell, the idea is that Neutron adds more information to a port object so that Nova does not need to make multiple calls to Neutron to collect all required networking information. It seems to have stalled for the time being, but bringing forward the edge computing use case might increase the interest again. Yes we've needed a sort of bulk query capability with the networking API for years. I started going down a path the other night of trying to see if we could bulk query floating IPs when building the internal instance network info cache [1] but it looks like that's not supported. The REST API docs for Neutron say that you can't OR filter query parameters together, but at the time looking at the code it seemed like it might be possible. Chris Friesen from Wind River has also been looking at some of this lately, see [2]. But getting people looking at doing performance profiling at scale and then identifying the major pain points would be really really helpful for the upstream development team that don't have access to those types of resources. Then we could prioritize investigating ways to fix those issues to improve performance. [1] https://review.openstack.org/#/c/465792/ [2] http://lists.openstack.org/pipermail/openstack-dev/2017-May/117096.html -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Massively distributed] Optimizing the interaction of OpenStack components
On 5/24/2017 6:48 AM, Ronan-Alexandre Cherrueau wrote: You can find examples of such diagrams that have been automatically generated on the website where we host results of our experiments[5]. This is nice, I've seen others do things like this before with Rally and osprofiler. The super thin vertical is hard to follow though, it would be nice if the graph could be expanded horizontally. -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Massively distributed] Optimizing the interaction of OpenStack components
Georg, > I unfortunately missed to join yesterday's working group meeting. Just > to confirm, the IRC meetings are every two weeks on Wednesday at 15 UTC. > Is that still correct? IRC meetings of Performance WG chaired by Dina Belova are every Tuesday, 15:30 UTC in `#openstack-performance'. IRC meetings of Fog Edge Massively Distributed Clouds WG chaired by Adrien Lebre and Anthony Simonet are every Wednesday of even weeks, 15 UTC in `#openstack-meeting'. So, next Fog Edge Massively Distributed Clouds WG IRC meeting starts in 1 hour and 20 minutes. Best regards, On Wed, May 24, 2017 at 2:16 PM, Georg Kunz wrote: > Hi Ronan, > >> First of all, sorry for the late response. I took a one-week vacation after >> the >> Summit to visit the US. > > No problem at all. I hope you had a great time. > >> > Nevertheless, one concrete thing which came to my mind, is this >> > proposed improvement of the interaction between Nova and Neutron: >> > https://review.openstack.org/#/c/390513/ >> > >> > In a nutshell, the idea is that Neutron adds more information to a >> > port object so that Nova does not need to make multiple calls to >> > Neutron to collect all required networking information. It seems to >> > have stalled for the time being, but bringing forward the edge >> > computing use case might increase the interest again. >> >> Thanks for pointing such amelioration. This is a good use case to start >> with. It >> should help to understand common patterns in the workflow that can be >> optimized. Let's see if we can implement an analysis with osp-utils that >> automatically highlight such pattern. > > Thanks a lot for the links. I'll dig into the tools to get a better > understanding and > make up my mind on how a potential analysis might look like. > > I unfortunately missed to join yesterday's working group meeting. Just to > confirm, > the IRC meetings are every two weeks on Wednesday at 15 UTC. Is that still > correct? > > Best regards > Georg > >> Any comments/remarks on how we can do that is more than welcome. >> >> Best regards, >> >> [1] https://github.com/BeyondTheClouds/enos >> [2] https://www.youtube.com/watch?v=xwT08H02Nok >> [3] https://osprofiler.readthedocs.io/ >> [4] https://github.com/BeyondTheClouds/osp-utils >> [5] http://enos.irisa.fr/html/seqdiag/v1/ > -- Ronan-A. Cherrueau https://rcherrueau.github.io __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Massively distributed] Optimizing the interaction of OpenStack components
Hi Ronan, > First of all, sorry for the late response. I took a one-week vacation after > the > Summit to visit the US. No problem at all. I hope you had a great time. > > Nevertheless, one concrete thing which came to my mind, is this > > proposed improvement of the interaction between Nova and Neutron: > > https://review.openstack.org/#/c/390513/ > > > > In a nutshell, the idea is that Neutron adds more information to a > > port object so that Nova does not need to make multiple calls to > > Neutron to collect all required networking information. It seems to > > have stalled for the time being, but bringing forward the edge > > computing use case might increase the interest again. > > Thanks for pointing such amelioration. This is a good use case to start with. > It > should help to understand common patterns in the workflow that can be > optimized. Let's see if we can implement an analysis with osp-utils that > automatically highlight such pattern. Thanks a lot for the links. I'll dig into the tools to get a better understanding and make up my mind on how a potential analysis might look like. I unfortunately missed to join yesterday's working group meeting. Just to confirm, the IRC meetings are every two weeks on Wednesday at 15 UTC. Is that still correct? Best regards Georg > Any comments/remarks on how we can do that is more than welcome. > > Best regards, > > [1] https://github.com/BeyondTheClouds/enos > [2] https://www.youtube.com/watch?v=xwT08H02Nok > [3] https://osprofiler.readthedocs.io/ > [4] https://github.com/BeyondTheClouds/osp-utils > [5] http://enos.irisa.fr/html/seqdiag/v1/ __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Massively distributed] Optimizing the interaction of OpenStack components
Hi Georg, First of all, sorry for the late response. I took a one-week vacation after the Summit to visit the US. > Hi Ronan, hi Adrien, > > we talked yesterday in the edge/massively distributed working group > meeting about optimizations of the control traffic between OpenStack > components. Besides investigating different message busses, we agreed > that it is also worth looking at how different OpenStack components > interact. My understanding is that you have already developed tracing > tools for this. Actually, we (Inria) developed a tool called EnOS[1] that lets us conduct performance analysis of different OpenStack deployments. Briefly, EnOS provides a simple topology definition and traffic shaping so you can deploy an OpenStack you envision, and then run benchmarking tools such as Rally (for control plane) and Shaker (for data plane) to measure OpenStack performance. We made a presentation during the Boston Summit that shows how we use EnOS to measure OpenStack performance in a Wide Area Network context[2]. During our experiments, most of the time we struggle to understand the workflow of OpenStack. Not everybody here is an expert in Nova plus Neutron plus Glance plus Keystone plus "whatever OpenStack services you consider in your deployment" so we add into EnOS the ability to produce OSProfiler traces[3] based on a Rally scenario. OSProfiler is an OpenStack profiling tool developed by folks of the performance team. Traces generated by OSProfiler give a fine-grained understanding of your workflow. Maybe too much fine-grained. Thus, we also developed a tool, called osp-utils[4], that enables us query an OSProfiler trace (filter, fold, count...) and automatically produce a sequence diagram to show interactions between services. You can find examples of such diagrams that have been automatically generated on the website where we host results of our experiments[5]. In these sequence diagrams, we choose to filter calls so that we only keep those that end by an RPC. This is why there are no interactions with Keystone or with the Database. But you can also choose to filter or fold on any information available in a call. Anyway, we believe that such view that clearly states interactions between services is helpful for the community. We also think that a query language for OSProfiler traces would help to run some analysis for the optimizations of the control traffic between OpenStack components. Please note that osp-utils tool is just a prototype right now -- something developed during some sleepless nights ;). So the code and documentation must be improved. > Nevertheless, one concrete thing which came to my mind, is this proposed > improvement of the interaction between Nova and Neutron: > https://review.openstack.org/#/c/390513/ > > In a nutshell, the idea is that Neutron adds more information to a port > object so that Nova does not need to make multiple calls to Neutron to > collect all required networking information. It seems to have stalled > for the time being, but bringing forward the edge computing use case > might increase the interest again. Thanks for pointing such amelioration. This is a good use case to start with. It should help to understand common patterns in the workflow that can be optimized. Let's see if we can implement an analysis with osp-utils that automatically highlight such pattern. Any comments/remarks on how we can do that is more than welcome. Best regards, [1] https://github.com/BeyondTheClouds/enos [2] https://www.youtube.com/watch?v=xwT08H02Nok [3] https://osprofiler.readthedocs.io/ [4] https://github.com/BeyondTheClouds/osp-utils [5] http://enos.irisa.fr/html/seqdiag/v1/ On Thu, May 11, 2017 at 8:44 PM, Georg Kunz wrote: > Hi Ronan, > > hi Adrien, > > > > we talked yesterday in the edge/massively distributed working group meeting > about optimizations of the control traffic between OpenStack components. > Besides investigating different message busses, we agreed that it is also > worth looking at how different OpenStack components interact. My > understanding is that you have already developed tracing tools for this. > > > > Nevertheless, one concrete thing which came to my mind, is this proposed > improvement of the interaction between Nova and Neutron: > > https://review.openstack.org/#/c/390513/ > > > > In a nutshell, the idea is that Neutron adds more information to a port > object so that Nova does not need to make multiple calls to Neutron to > collect all required networking information. It seems to have stalled for > the time being, but bringing forward the edge computing use case might > increase the interest again. > > > > Best regards > > Georg > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Ronan-A. Cherr
[openstack-dev] [Massively distributed] Optimizing the interaction of OpenStack components
Hi Ronan, hi Adrien, we talked yesterday in the edge/massively distributed working group meeting about optimizations of the control traffic between OpenStack components. Besides investigating different message busses, we agreed that it is also worth looking at how different OpenStack components interact. My understanding is that you have already developed tracing tools for this. Nevertheless, one concrete thing which came to my mind, is this proposed improvement of the interaction between Nova and Neutron: https://review.openstack.org/#/c/390513/ In a nutshell, the idea is that Neutron adds more information to a port object so that Nova does not need to make multiple calls to Neutron to collect all required networking information. It seems to have stalled for the time being, but bringing forward the edge computing use case might increase the interest again. Best regards Georg __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
