Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
On Wed, Jul 2, 2014 at 5:46 AM, Ihar Hrachyshka wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > On 01/07/14 17:14, Alexei Kornienko wrote: >> Hi, >> >> Please see some minor comments inline. Do you think we can schedule >> some time to discuss this topic on one of the upcoming meetings? We >> can come out with some kind of the summary and actions plan to >> start working on. > > Please check: https://wiki.openstack.org/wiki/Meetings/Oslo I think > you can add your case in the agenda. That's how it works in e.g. Neutron. I've been trying to keep the Oslo meetings focused on status updates and issue reports from or liaisons. Let's work on reaching consensus on this change here on the mailing list where we have the time and space for a long-form conversation. Doug > There are a lot of questions that should be answered to implement this: Where such tests would run (jenking, local PC, devstack VM)? > I would expect it to be exposed to jenkins thru 'tox'. We > then can set up a separate job to run them and compare with a > base line [TBD: what *is* baseline?] to make sure we don't > introduce performance regressions. >> Such tests cannot be exposed thru 'tox' since they require >> some environment setup (rabbitmq-server, zeromq matchmaker, >> etc.). Such setup is way out of scope for tox. Cause of >> this we should find some other way to run such tests. >> You may just assume server is already set and available thru a >> common socket. >>> Assuming that something is already setup is not an option. If we >>> add new env to tox I assume that anyone can run it locally and >>> get the same results as he would get from jenkins. > > Fair enough. You can look into tempest then. I don't know whether they > already have some framework for performance testing though. > > /Ihar > -BEGIN PGP SIGNATURE- > Version: GnuPG/MacGPG2 v2.0.22 (Darwin) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQEcBAEBCgAGBQJTs9UIAAoJEC5aWaUY1u57USMH/j4dfCdAl2QKCMLDWzopLqon > 41gqq8DsKDhc/I24z2KNYqRPk43tQvR9dWhbnbs1XlqXk0keozQ2YNE8pvuZyqok > VN4/l435ewHJ3OyZePQLANgDBrAVdWRmmLWJwGJZ8ceTfZRV1MN+vsLHGwCMFinU > J/8UAR8XUF5UVrd/VONsvVSwDnx1v7vs8zM9aICfos0F4ByMU9bPThdeUiuJOTuU > xNP+FEdlNwwlc4sAwm3qHKJIVe7gYSojIfQGhmBlXWpPKA1SYoT5I6qMZibsbtlM > VE092du9APc8LLnE3DdMS6CuUuABbljUtAxUr5z8CirIDoh9n/c5+e8Jfbb8zHU= > =OwOm > -END PGP SIGNATURE- > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
On 07/01/2014 04:14 PM, Alexei Kornienko wrote: A lot of driver details leak outside the API and it makes it hard to improve driver without changing the API. I agree that some aspects of specific driver implementations leak into the public API for the messaging library as a whole. There are also some ambiguities around the intended and actual contracts. However, I do also think that the existing driver abstraction, while not perfect, can be used to implement drivers using quite different approaches. The current impl_rabbit and impl_qpid drivers may be hard(er) to change, but that's not because of the API (in my view). [...] This would allow us to change drivers without touching the API and test their performance separately I think you can do this already. It is true that there are some semantic differences when switching drivers, and it would be ideal to minimise those (or at least make them more explicit) at some point. I did some experiments measuring the scalability of rpc calls for different drivers - mainly as an initial stress test for the proposed AMQP 1.0 driver -which showed significant differences even where the same broker was used, just by changing the driver. I'm not arguing that there will never be a good reason to change either the driver API or indeed the public API, but I think a lot can be accomplished within the current framework if desired. (With more specific changes to APIs then proposed with some context as needed). --Gordon ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
On 07/01/2014 03:52 PM, Ihar Hrachyshka wrote: On 01/07/14 15:55, Alexei Kornienko wrote: And in addition I've provided some links to existing implementation with places that IMHO cause bottlenecks. From my point of view that code is doing obviously stupid things (like closing/opening sockets for each message sent). That indeed sounds bad. That is enough for me to rewrite it even without additional proofs that it's wrong. [Full disclosure: I'm not as involved into oslo.messaging internals as you probably are, so I may speak out dumb things.] I wonder whether there are easier ways to fix that particular issue without rewriting everything from scratch. Like, provide a pool of connections and make send() functions use it instead of creating new connections (?) I did a little investigation, using the test scripts from Alexei. Looking at the protocol trace with wireshark, you can certainly see lots of redundant stuff. However I did *not* see connections being opened and closed. What I did see was the AMQP (0-9-1) channel being opened and closed, and the exchange being declared for every published message. Both these actions are synchronous. This clearly impacts throughput and, in my view more importantly, will limit scalability by giving the broker redundant work. However, I think this could be avoided (at least to the extreme extent it occurs at present) by a simple patch. I posted this up to gerrit in case anyone is interested: https://review.openstack.org/#/c/104194/ With this in place the ingress improves significantly[1]. The bottleneck then appears to be the acking of messages. This means the queue depth tends to get higher than before, and the traffic appears more 'bursty'. Each message is acked individually, whereas some degree of batching might be possible which would likely improve the efficiency there a little bit. Anyway, just an observation I thought it would be interesting to share. --Gordon. [1] From less than 400 msgs/sec to well over 1000 msgs/sec on my laptop ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 01/07/14 17:14, Alexei Kornienko wrote: > Hi, > > Please see some minor comments inline. Do you think we can schedule > some time to discuss this topic on one of the upcoming meetings? We > can come out with some kind of the summary and actions plan to > start working on. Please check: https://wiki.openstack.org/wiki/Meetings/Oslo I think you can add your case in the agenda. That's how it works in e.g. Neutron. >>> There are a lot of questions that should be answered >>> to implement this: Where such tests would run (jenking, >>> local PC, devstack VM)? I would expect it to be exposed to jenkins thru 'tox'. We then can set up a separate job to run them and compare with a base line [TBD: what *is* baseline?] to make sure we don't introduce performance regressions. > Such tests cannot be exposed thru 'tox' since they require > some environment setup (rabbitmq-server, zeromq matchmaker, > etc.). Such setup is way out of scope for tox. Cause of > this we should find some other way to run such tests. > You may just assume server is already set and available thru a > common socket. >> Assuming that something is already setup is not an option. If we >> add new env to tox I assume that anyone can run it locally and >> get the same results as he would get from jenkins. Fair enough. You can look into tempest then. I don't know whether they already have some framework for performance testing though. /Ihar -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCgAGBQJTs9UIAAoJEC5aWaUY1u57USMH/j4dfCdAl2QKCMLDWzopLqon 41gqq8DsKDhc/I24z2KNYqRPk43tQvR9dWhbnbs1XlqXk0keozQ2YNE8pvuZyqok VN4/l435ewHJ3OyZePQLANgDBrAVdWRmmLWJwGJZ8ceTfZRV1MN+vsLHGwCMFinU J/8UAR8XUF5UVrd/VONsvVSwDnx1v7vs8zM9aICfos0F4ByMU9bPThdeUiuJOTuU xNP+FEdlNwwlc4sAwm3qHKJIVe7gYSojIfQGhmBlXWpPKA1SYoT5I6qMZibsbtlM VE092du9APc8LLnE3DdMS6CuUuABbljUtAxUr5z8CirIDoh9n/c5+e8Jfbb8zHU= =OwOm -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Hi, Please see some minor comments inline. Do you think we can schedule some time to discuss this topic on one of the upcoming meetings? We can come out with some kind of the summary and actions plan to start working on. Regards, On 07/01/2014 05:52 PM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 01/07/14 15:55, Alexei Kornienko wrote: Hi, Thanks for detailed answer. Please see my comments inline. Regards, On 07/01/2014 04:28 PM, Ihar Hrachyshka wrote: On 30/06/14 21:34, Alexei Kornienko wrote: Hello, My understanding is that your analysis is mostly based on running a profiler against the code. Network operations can be bottlenecked in other places. You compare 'simple script using kombu' with 'script using oslo.messaging'. You don't compare script using oslo.messaging before refactoring and 'after that. The latter would show whether refactoring was worth the effort. Your test shows that oslo.messaging performance sucks, but it's not definite that hotspots you've revealed, once fixed, will show huge boost. My concern is that it may turn out that once all the effort to refactor the code is done, we won't see major difference. So we need base numbers, and performance tests would be a great helper here. It's really sad for me to see so little faith in what I'm saying. The test I've done using plain kombu driver was needed exactly to check that network is not the bottleneck for messaging performance. If you don't believe in my performance analysis we could ask someone else to do their own research and provide results. Technology is not about faith. :) First, let me make it clear I'm *not* against refactoring or anything that will improve performance. I'm just a bit skeptical, but hopefully you'll be able to show everyone I'm wrong, and then the change will occur. :) To add more velocity to your effort, strong arguments should be present. To facilitate that, I would start from adding performance tests that would give us some basis for discussion of changes proposed later. Please see below for detailed answer about performance tests implementation. It explains a bit why it's hard to present arguments that would be strong enough for you. I may run performance tests locally but it's not enough for community. Yes, that's why shipping some tests ready to run with oslo.messaging can help. Science is about reproducility, right? ;) And in addition I've provided some links to existing implementation with places that IMHO cause bottlenecks. From my point of view that code is doing obviously stupid things (like closing/opening sockets for each message sent). That indeed sounds bad. That is enough for me to rewrite it even without additional proofs that it's wrong. [Full disclosure: I'm not as involved into oslo.messaging internals as you probably are, so I may speak out dumb things.] I wonder whether there are easier ways to fix that particular issue without rewriting everything from scratch. Like, provide a pool of connections and make send() functions use it instead of creating new connections (?) I've tried to find a way to fix that without big changes but unfortunately I've failed to do so. Problem I see is that connection pool is defined and used on 1 layer of library and the problem is on the other. To fix this issues we need to change several layers of code and it's shared between 2 drivers - rabbit, qpid. Cause of this it seems really hard to make some logically finished and working patches that would allow us to move in proper direction without big refactoring of the drivers structure. Then, describing proposed details in a spec will give more exposure to your ideas. At the moment, I see general will to enhance the library, but not enough details on how to achieve this. Specification can make us think not about the burden of change that obviously makes people skeptic about rewrite-all approach, but about specific technical issues. I agree that we should start with a spec. However instead of having spec of needed changes I would prefer to have a spec describing needed functionality of the library (it may differ from existing functionality). Meaning, breaking API, again? It's not about breaking the API it's about making it more logical and independent. Right now it's not clear to me what API classes are used and how they are used. A lot of driver details leak outside the API and it makes it hard to improve driver without changing the API. What I would like to see is a clear definition of what library should provide and API interface that it should implement. It may be a little bit java like so API should be defined and frozed and anyone could propose their driver implementation using kombu/qpid/zeromq or pigeons and trained dolphins to deliver messages. This would allow us to change drivers without touching the API and test their performance separately. Using such a spec we could decide what it needed and what needs to be removed to
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 01/07/14 15:55, Alexei Kornienko wrote: > Hi, > > Thanks for detailed answer. Please see my comments inline. > > Regards, > > On 07/01/2014 04:28 PM, Ihar Hrachyshka wrote: On 30/06/14 21:34, > Alexei Kornienko wrote: Hello, My understanding is that your analysis is mostly based on running a profiler against the code. Network operations can be bottlenecked in other places. You compare 'simple script using kombu' with 'script using oslo.messaging'. You don't compare script using oslo.messaging before refactoring and 'after that. The latter would show whether refactoring was worth the effort. Your test shows that oslo.messaging performance sucks, but it's not definite that hotspots you've revealed, once fixed, will show huge boost. My concern is that it may turn out that once all the effort to refactor the code is done, we won't see major difference. So we need base numbers, and performance tests would be a great helper here. It's really sad for me to see so little faith in what I'm saying. The test I've done using plain kombu driver was needed exactly to check that network is not the bottleneck for messaging performance. If you don't believe in my performance analysis we could ask someone else to do their own research and provide results. > Technology is not about faith. :) > > First, let me make it clear I'm *not* against refactoring or > anything that will improve performance. I'm just a bit skeptical, > but hopefully you'll be able to show everyone I'm wrong, and then > the change will occur. :) > > To add more velocity to your effort, strong arguments should be > present. To facilitate that, I would start from adding performance > tests that would give us some basis for discussion of changes > proposed later. >> Please see below for detailed answer about performance tests >> implementation. It explains a bit why it's hard to present >> arguments that would be strong enough for you. I may run >> performance tests locally but it's not enough for community. Yes, that's why shipping some tests ready to run with oslo.messaging can help. Science is about reproducility, right? ;) > >> And in addition I've provided some links to existing >> implementation with places that IMHO cause bottlenecks. From my >> point of view that code is doing obviously stupid things (like >> closing/opening sockets for each message sent). That indeed sounds bad. >> That is enough for me to rewrite it even without additional >> proofs that it's wrong. [Full disclosure: I'm not as involved into oslo.messaging internals as you probably are, so I may speak out dumb things.] I wonder whether there are easier ways to fix that particular issue without rewriting everything from scratch. Like, provide a pool of connections and make send() functions use it instead of creating new connections (?) > > Then, describing proposed details in a spec will give more exposure > to your ideas. At the moment, I see general will to enhance the > library, but not enough details on how to achieve this. > Specification can make us think not about the burden of change that > obviously makes people skeptic about rewrite-all approach, but > about specific technical issues. >> I agree that we should start with a spec. However instead of >> having spec of needed changes I would prefer to have a spec >> describing needed functionality of the library (it may differ >> from existing functionality). Meaning, breaking API, again? >> Using such a spec we could decide what it needed and what needs >> to be removed to achieve what we need. > Problem with refactoring that I'm planning is that it's not a minor refactoring that can be applied in one patch but it's the whole library rewritten from scratch. > You can still maintain a long sequence of patches, like we did when > we migrated neutron to oslo.messaging (it was like ~25 separate > pieces). >> Talking into account possible gate issues I would like to avoid >> long series of patches since they won't be able to land at the >> same time and rebasing will become a huge pain. But you're the one proposing the change, you need to take burden. Having a new branch for everything-rewritten version of the library means that each bug fix or improvement to the library will require being tracked by each developer in two branches, with significantly different code. I think it's more honest to put rebase pain on people who rework the code than on everyone else. >> If we decide to start working on 2.0 API/implementation I think a >> topic branch 2.0a would be much better. I respectfully disagree. See above. > Existing messaging code was written long long time ago (in a galaxy far far away maybe?) and it was copy-pasted directly from nova. It was not built as a library and it was never
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Hi, Thanks for detailed answer. Please see my comments inline. Regards, On 07/01/2014 04:28 PM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 30/06/14 21:34, Alexei Kornienko wrote: Hello, My understanding is that your analysis is mostly based on running a profiler against the code. Network operations can be bottlenecked in other places. You compare 'simple script using kombu' with 'script using oslo.messaging'. You don't compare script using oslo.messaging before refactoring and 'after that. The latter would show whether refactoring was worth the effort. Your test shows that oslo.messaging performance sucks, but it's not definite that hotspots you've revealed, once fixed, will show huge boost. My concern is that it may turn out that once all the effort to refactor the code is done, we won't see major difference. So we need base numbers, and performance tests would be a great helper here. It's really sad for me to see so little faith in what I'm saying. The test I've done using plain kombu driver was needed exactly to check that network is not the bottleneck for messaging performance. If you don't believe in my performance analysis we could ask someone else to do their own research and provide results. Technology is not about faith. :) First, let me make it clear I'm *not* against refactoring or anything that will improve performance. I'm just a bit skeptical, but hopefully you'll be able to show everyone I'm wrong, and then the change will occur. :) To add more velocity to your effort, strong arguments should be present. To facilitate that, I would start from adding performance tests that would give us some basis for discussion of changes proposed later. Please see below for detailed answer about performance tests implementation. It explains a bit why it's hard to present arguments that would be strong enough for you. I may run performance tests locally but it's not enough for community. And in addition I've provided some links to existing implementation with places that IMHO cause bottlenecks. From my point of view that code is doing obviously stupid things (like closing/opening sockets for each message sent). That is enough for me to rewrite it even without additional proofs that it's wrong. Then, describing proposed details in a spec will give more exposure to your ideas. At the moment, I see general will to enhance the library, but not enough details on how to achieve this. Specification can make us think not about the burden of change that obviously makes people skeptic about rewrite-all approach, but about specific technical issues. I agree that we should start with a spec. However instead of having spec of needed changes I would prefer to have a spec describing needed functionality of the library (it may differ from existing functionality). Using such a spec we could decide what it needed and what needs to be removed to achieve what we need. Problem with refactoring that I'm planning is that it's not a minor refactoring that can be applied in one patch but it's the whole library rewritten from scratch. You can still maintain a long sequence of patches, like we did when we migrated neutron to oslo.messaging (it was like ~25 separate pieces). Talking into account possible gate issues I would like to avoid long series of patches since they won't be able to land at the same time and rebasing will become a huge pain. If we decide to start working on 2.0 API/implementation I think a topic branch 2.0a would be much better. Existing messaging code was written long long time ago (in a galaxy far far away maybe?) and it was copy-pasted directly from nova. It was not built as a library and it was never intended to be used outside of nova. Some parts of it cannot even work normally cause it was not designed to work with drivers like zeromq (matchmaker stuff). oslo.messaging is NOT the code you can find in oslo-incubator rpc module. It was hugely rewritten to expose a new, cleaner API. This is btw one of the reasons migration to this new library is so painful. It was painful to move to oslo.messaging, so we need clear need for a change before switching to yet another library. API indeed has changed but general implementation details and processing flow goes way back to 2011 and nova code (for example general Publisher/Consumer implementation in impl_rabbit) That's the code I'm talking about. Refactoring as I see it will do the opposite thing. It will keep intact as much API as possible but change internals to make it more efficient (that's why I call it refactoring) So 2.0 version might be (partially?) backwards compatible and migration won't be such a pain. The reason I've raised this question on the mailing list was to get some agreement about future plans of oslo.messaging development and start working on it in coordination with community. For now I don't see any actions plan emerging from it. I would like to see us bringing more co
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 30/06/14 21:34, Alexei Kornienko wrote: > Hello, > > > My understanding is that your analysis is mostly based on running > a profiler against the code. Network operations can be bottlenecked > in other places. > > You compare 'simple script using kombu' with 'script using > oslo.messaging'. You don't compare script using oslo.messaging > before refactoring and 'after that. The latter would show whether > refactoring was worth the effort. Your test shows that > oslo.messaging performance sucks, but it's not definite that > hotspots you've revealed, once fixed, will show huge boost. > > My concern is that it may turn out that once all the effort to > refactor the code is done, we won't see major difference. So we > need base numbers, and performance tests would be a great helper > here. > > > It's really sad for me to see so little faith in what I'm saying. > The test I've done using plain kombu driver was needed exactly to > check that network is not the bottleneck for messaging > performance. If you don't believe in my performance analysis we > could ask someone else to do their own research and provide > results. Technology is not about faith. :) First, let me make it clear I'm *not* against refactoring or anything that will improve performance. I'm just a bit skeptical, but hopefully you'll be able to show everyone I'm wrong, and then the change will occur. :) To add more velocity to your effort, strong arguments should be present. To facilitate that, I would start from adding performance tests that would give us some basis for discussion of changes proposed later. Then, describing proposed details in a spec will give more exposure to your ideas. At the moment, I see general will to enhance the library, but not enough details on how to achieve this. Specification can make us think not about the burden of change that obviously makes people skeptic about rewrite-all approach, but about specific technical issues. > > Problem with refactoring that I'm planning is that it's not a > minor refactoring that can be applied in one patch but it's the > whole library rewritten from scratch. You can still maintain a long sequence of patches, like we did when we migrated neutron to oslo.messaging (it was like ~25 separate pieces). > Existing messaging code was written long long time ago (in a galaxy > far far away maybe?) and it was copy-pasted directly from nova. It > was not built as a library and it was never intended to be used > outside of nova. Some parts of it cannot even work normally cause > it was not designed to work with drivers like zeromq (matchmaker > stuff). oslo.messaging is NOT the code you can find in oslo-incubator rpc module. It was hugely rewritten to expose a new, cleaner API. This is btw one of the reasons migration to this new library is so painful. It was painful to move to oslo.messaging, so we need clear need for a change before switching to yet another library. > > The reason I've raised this question on the mailing list was to get > some agreement about future plans of oslo.messaging development and > start working on it in coordination with community. For now I don't > see any actions plan emerging from it. I would like to see us > bringing more constructive ideas about what should be done. > > If you think that first action should be profiling lets discuss how > it should be implemented (cause it works for me just fine on my > local PC). I guess we'll need to define some basic scenarios that > would show us overall performance of the library. Let's start from basic send/receive throughput, for tiny and large messages, multiple consumers etc. > There are a lot of questions that should be answered to implement > this: Where such tests would run (jenking, local PC, devstack VM)? I would expect it to be exposed to jenkins thru 'tox'. We then can set up a separate job to run them and compare with a base line [TBD: what *is* baseline?] to make sure we don't introduce performance regressions. > How such scenarios should look like? How do we measure performance > (cProfile, etc.)? I think we're interested in message rate, not CPU utilization. > How do we collect results? How do we analyze results to find > bottlenecks? etc. > > Another option would be to spend some of my free time implementing > mentioned refactoring (as I see it) and show you the results of > performance testing compared with existing code. This approach generally doesn't work beyond PoC. Openstack is a complex project, and we need to stick to procedures - spec review, then coding, all in upstream, with no private branches outside common infrastructure. > The only problem with such approach is that my code won't be > oslo.messaging and it won't be accepted by community. It may be > drop in base for v2.0 but I'm afraid this won't be acceptable > either. > Future does not occur here that way. If you want your work to be consumed by community,
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
I'm far from an oslo.messaging expert, but a few general thoughts below. On 06/30/2014 02:34 PM, Alexei Kornienko wrote: > Hello, > > >> My understanding is that your analysis is mostly based on running a >> profiler against the code. Network operations can be bottlenecked in >> other places. >> >> You compare 'simple script using kombu' with 'script using >> oslo.messaging'. You don't compare script using oslo.messaging before >> refactoring and 'after that. The latter would show whether refactoring >> was worth the effort. Your test shows that oslo.messaging performance >> sucks, but it's not definite that hotspots you've revealed, once >> fixed, will show huge boost. >> >> My concern is that it may turn out that once all the effort to >> refactor the code is done, we won't see major difference. So we need >> base numbers, and performance tests would be a great helper here. >> > > It's really sad for me to see so little faith in what I'm saying. > The test I've done using plain kombu driver was needed exactly to check > that network is not the bottleneck for messaging performance. > If you don't believe in my performance analysis we could ask someone else > to do their own research and provide results. The problem is that extremely simple test cases are often not representative of overall performance, so comparing a purpose-built test doing a single thing as fast as possible to a full library that has to be able to handle all of OpenStack's messaging against every supported back end isn't sufficient on its own to convince me that there is a "rewrite all the things" issue here. > > Problem with refactoring that I'm planning is that it's not a minor > refactoring that can be applied in one patch but it's the whole library > rewritten from scratch. Which is exactly why we want to make sure it's something that needs to be done before heading down that path. I know I've wasted more time than I'd like to admit optimizing the wrong code paths, only to find that my changes made a .1% difference because I was mistaken about what the bottleneck was. Add to that the fact that we're _just_ completing the migration to oslo.messaging in the first place and I hope you can understand why no one wants to undertake another massive, possibly compatibility breaking, refactoring unless we're absolutely certain it's the only way to address the performance limitations of the existing code. > Existing messaging code was written long long time ago (in a galaxy far far > away maybe?) and it was copy-pasted directly from nova. > It was not built as a library and it was never intended to be used outside > of nova. This isn't really true anymore. The oslo.messaging code underwent significant changes in the move from the incubator rpc module to the oslo.messaging library. One of the major points of emphasis in all Oslo graduations is to make sure the new lib has a proper API and isn't just a naive copy-paste of the existing code. > Some parts of it cannot even work normally cause it was not designed to > work with drivers like zeromq (matchmaker stuff). > > The reason I've raised this question on the mailing list was to get some > agreement about future plans of oslo.messaging development and start > working on it in coordination with community. > For now I don't see any actions plan emerging from it. I would like to see > us bringing more constructive ideas about what should be done. > > If you think that first action should be profiling lets discuss how it > should be implemented (cause it works for me just fine on my local PC). > I guess we'll need to define some basic scenarios that would show us > overall performance of the library. > There are a lot of questions that should be answered to implement this: > Where such tests would run (jenking, local PC, devstack VM)? > How such scenarios should look like? > How do we measure performance (cProfile, etc.)? > How do we collect results? > How do we analyze results to find bottlenecks? > etc. > > Another option would be to spend some of my free time implementing > mentioned refactoring (as I see it) and show you the results of performance > testing compared with existing code. > The only problem with such approach is that my code won't be oslo.messaging > and it won't be accepted by community. It may be drop in base for v2.0 but > I'm afraid this won't be acceptable either. > > Regards, > Alexei Kornienko > > > 2014-06-30 17:51 GMT+03:00 Gordon Sim : > >> On 06/30/2014 12:22 PM, Ihar Hrachyshka wrote: >> >> Alexei Kornienko wrote: > Some performance tests may be introduced but they would be more > like functional tests since they require setup of actual > messaging server (rabbit, etc.). > >>> Yes. I think we already have some. F.e. >>> tests/drivers/test_impl_qpid.py attempts to use local Qpid server >>> (backing up to fake server if it's not available). >>> >> >> I always get failures when there is a real qpidd service listening on t
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Hello, > My understanding is that your analysis is mostly based on running a > profiler against the code. Network operations can be bottlenecked in > other places. > > You compare 'simple script using kombu' with 'script using > oslo.messaging'. You don't compare script using oslo.messaging before > refactoring and 'after that. The latter would show whether refactoring > was worth the effort. Your test shows that oslo.messaging performance > sucks, but it's not definite that hotspots you've revealed, once > fixed, will show huge boost. > > My concern is that it may turn out that once all the effort to > refactor the code is done, we won't see major difference. So we need > base numbers, and performance tests would be a great helper here. > It's really sad for me to see so little faith in what I'm saying. The test I've done using plain kombu driver was needed exactly to check that network is not the bottleneck for messaging performance. If you don't believe in my performance analysis we could ask someone else to do their own research and provide results. Problem with refactoring that I'm planning is that it's not a minor refactoring that can be applied in one patch but it's the whole library rewritten from scratch. Existing messaging code was written long long time ago (in a galaxy far far away maybe?) and it was copy-pasted directly from nova. It was not built as a library and it was never intended to be used outside of nova. Some parts of it cannot even work normally cause it was not designed to work with drivers like zeromq (matchmaker stuff). The reason I've raised this question on the mailing list was to get some agreement about future plans of oslo.messaging development and start working on it in coordination with community. For now I don't see any actions plan emerging from it. I would like to see us bringing more constructive ideas about what should be done. If you think that first action should be profiling lets discuss how it should be implemented (cause it works for me just fine on my local PC). I guess we'll need to define some basic scenarios that would show us overall performance of the library. There are a lot of questions that should be answered to implement this: Where such tests would run (jenking, local PC, devstack VM)? How such scenarios should look like? How do we measure performance (cProfile, etc.)? How do we collect results? How do we analyze results to find bottlenecks? etc. Another option would be to spend some of my free time implementing mentioned refactoring (as I see it) and show you the results of performance testing compared with existing code. The only problem with such approach is that my code won't be oslo.messaging and it won't be accepted by community. It may be drop in base for v2.0 but I'm afraid this won't be acceptable either. Regards, Alexei Kornienko 2014-06-30 17:51 GMT+03:00 Gordon Sim : > On 06/30/2014 12:22 PM, Ihar Hrachyshka wrote: > > Alexei Kornienko wrote: >>> Some performance tests may be introduced but they would be more like functional tests since they require setup of actual messaging server (rabbit, etc.). >>> >> Yes. I think we already have some. F.e. >> tests/drivers/test_impl_qpid.py attempts to use local Qpid server >> (backing up to fake server if it's not available). >> > > I always get failures when there is a real qpidd service listening on the > expected port. Does anyone else see this? > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
On 06/30/2014 12:22 PM, Ihar Hrachyshka wrote: Alexei Kornienko wrote: Some performance tests may be introduced but they would be more like functional tests since they require setup of actual messaging server (rabbit, etc.). Yes. I think we already have some. F.e. tests/drivers/test_impl_qpid.py attempts to use local Qpid server (backing up to fake server if it's not available). I always get failures when there is a real qpidd service listening on the expected port. Does anyone else see this? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 > On 06/27/2014 04:04 PM, Ihar Hrachyshka wrote: On 26/06/14 22:38, > Alexei Kornienko wrote: Hello Jay, Benchmark for oslo.messaging is really simple: You create a client that sends messages infinitively and a server that processes them. After you can use rabbitmq management plugin to see average throughput in the queue. Simple example can be found here - https://github.com/andreykurilin/profiler-for-oslo-messaging I've mentioned some of this already in my previous mails but here it is again: > "Huge" is descriptive but not quantitative :) Do you have > any numbers that pinpoint the amount of time that is being > spent reconstructing and declaring the queues, say, > compared to the time spent doing transmission? I don't have precise slowdown percentage for each issue. I've just identified hotspots using cProfile and strace. > This ten times number... is that an estimate or do you have > hard numbers that show that? Just curious. Using a script that I've mentioned earlier I get average throughput on my PC ~700 cast calls per second. I've written a simple and stupid script that uses kombu directly (in a single threaded and synchronous way with single connection) It gave me throughput ~8000 messages per second. Thats why I say that library should work at least 10 times faster. > It doesn't show that those major issues you've pointed out result > in such large message processing speed dropdown though. Maybe there > are other causes of slowdown we observe. Neither it shows that > refactoring of the code will actually help to boost the library > significantly. >> It doesn't show that those major issues are the *only* reason for >> slowdown. Bit it shows that those issues are biggest that are >> currently visible. My understanding is that your analysis is mostly based on running a profiler against the code. Network operations can be bottlenecked in other places. You compare 'simple script using kombu' with 'script using oslo.messaging'. You don't compare script using oslo.messaging before refactoring and 'after that. The latter would show whether refactoring was worth the effort. Your test shows that oslo.messaging performance sucks, but it's not definite that hotspots you've revealed, once fixed, will show huge boost. My concern is that it may turn out that once all the effort to refactor the code is done, we won't see major difference. So we need base numbers, and performance tests would be a great helper here. >> We'll get a major speed boost if we fix them (and possibly >> discover new issues the would prevent us of reaching full >> speed). > > Though having some hard numbers from using kombu directly is still > a good thing. Is it possible that we introduce some performance > tests into oslo.messaging itself? >> Some performance tests may be introduced but they would be more >> like functional tests since they require setup of actual >> messaging server (rabbit, etc.). Yes. I think we already have some. F.e. tests/drivers/test_impl_qpid.py attempts to use local Qpid server (backing up to fake server if it's not available). We could create a separate subtree in the library for functional tests. >> What do you mean exactly by performance tests? Just testing >> overall throughput with some basic scenarios or you mean finding >> hotspots in the code? > The former. Once we have some base numbers, we may use them to consider whether changes you propose are worth the effort. > Regards, Alexei Kornienko On 06/26/2014 11:22 PM, Jay Pipes wrote: > Hey Alexei, thanks for sharing your findings. Comments > inline. > > On 06/26/2014 10:08 AM, Alexei Kornienko wrote: >> Hello, >> >> Returning to performance issues of oslo.messaging. I've >> found 2 biggest issue in existing implementation of >> rabbit (kombu) driver: >> >> 1) For almost every message sent/received a new object >> of Consumer/Publisher class is created. And each object >> of this class tries to declare it's queue even if it's >> already declared. >> https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L159 >> >> >> >> >> > >> This causes a huge slowdown. > "Huge" is descriptive but not quantitative :) Do you have > any numbers that pinpoint the amount of time that is being > spent reconstructing and declaring the queues, say, > compared to the time spent doing transmission? > >> 2) with issue #1 is fixed (I've applied a small hack to >> fix it in my repo) the next big issue araise. For every >> rpc message received a reply is sent when processing is >> done (it seems that reply is sent even for "cast" calls >> which it really strange to me). Reply sent is using >> connect
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
On 6/27/2014 11:27 AM, Alexei Kornienko wrote: Hi, Why should we create queue in advance? Notifications are used for communicating with downstream systems (which may or may not be online at the time). This includes dashboards, monitoring systems, billing systems, etc. They can't afford to lose these important updates. So, a queue has to exist and the events just build-up until they are eaten. RPC doesn't need this though. Let's consider following use cases: 1) * listener starts and creates a queue * publishers connect to exchange and start publishing No need to create a queue in advance here since listener does it when it starts Right, this is the RPC case. 2) * publishers create a queue in advance and start publishing Creation is not correct since there is no guarantee that someone would ever use this queue... This is why notifications are turned off by default. IMHO listener should create a queue and publishers should not care about it at all. What do you think? See above. There are definite use-cases where the queue has to be created in advance. But, as I say, RPC isn't one of them. So, for 90% of the AMQP traffic, we don't need this feature. We should be able to disable it for RPC in oslo.messaging. (I say "should" because I'm not positive some aspect of openstack doesn't depend on the queue existing. Thinking about the scheduler mostly) -S On 06/27/2014 05:16 PM, Sandy Walsh wrote: Something to consider is the "create the queue in advance" feature is done for notifications, so we don't drop important messages on the floor by having an Exchange with no associated Queue. For RPC operations, this may not be required (we assume the service is available). If this check is truly a time-sink we could ignore that check for rpc calls. -S On 6/10/2014 9:31 AM, Alexei Kornienko wrote: Hi, Please find some answers inline. Regards, Alexei On 06/10/2014 03:06 PM, Flavio Percoco wrote: On 10/06/14 15:03 +0400, Dina Belova wrote: Hello, stackers! Oslo.messaging is future of how different OpenStack components communicate with each other, and really I’d love to start discussion about how we can make this library even better then it’s now and how can we refactor it make more production-ready. As we all remember, oslo.messaging was initially inspired to be created as a logical continuation of nova.rpc - as a separated library, with lots of transports supported, etc. That’s why oslo.messaging inherited not only advantages of now did the nova.rpc work (and it were lots of them), but also some architectural decisions that currently sometimes lead to the performance issues (we met some of them while Ceilometer performance testing [1] during the Icehouse). For instance, simple testing messaging server (with connection pool and eventlet) can process 700 messages per second. The same functionality implemented using plain kombu (without connection pool and eventlet) driver is processing ten times more - 7000-8000 messages per second. So we have the following suggestions about how we may make this process better and quicker (and really I’d love to collect your feedback, folks): 1) Currently we have main loop running in the Executor class, and I guess it’ll be much better to move it to the Server class, as it’ll make relationship between the classes easier and will leave Executor only one task - process the message and that’s it (in blocking or eventlet mode). Moreover, this will make further refactoring much easier. To some extent, the executors are part of the server class since the later is the one actually controlling them. If I understood your proposal, the server class would implement the event loop, which means we would have an EventletServer / BlockingServer, right? If what I said is what you meant, then I disagree. Executors keep the eventloop isolated from other parts of the library and this is really important for us. One of the reason is to easily support multiple python versions - by having different event loops. Is my assumption correct? Could you elaborate more? No It's not how we plan it. Server will do the loop and pass received message to dispatcher and executor. It means that we would still have blocking executor and eventlet executor in the same server class. We would just change the implementation part to make it more consistent and easier to control. 2) Some of the drivers implementations (such as impl_rabbit and impl_qpid, for instance) are full of useless separated classes that in reality might be included to other ones. There are already some changes making the whole structure easier [2], and after the 1st issue will be solved Dispatcher and Listener also will be able to be refactored. This was done on purpose. The idea was to focus on backwards compatibility rather than cleaning up/improving the drivers. That said, sounds like those drivers could user some clean up. However, I think we should first extend the test suite a bit more before ha
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
On 06/26/2014 09:38 PM, Alexei Kornienko wrote: Benchmark for oslo.messaging is really simple: You create a client that sends messages infinitively and a server that processes them. After you can use rabbitmq management plugin to see average throughput in the queue. Simple example can be found here - https://github.com/andreykurilin/profiler-for-oslo-messaging Just fyi, I also recently did some testing, though focused on scaling of rpc calls rather than throughput of a single client-server pair. The test I used is here in case anyone is interested: https://github.com/grs/ombt. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Hi, Why should we create queue in advance? Let's consider following use cases: 1) * listener starts and creates a queue * publishers connect to exchange and start publishing No need to create a queue in advance here since listener does it when it starts 2) * publishers create a queue in advance and start publishing Creation is not correct since there is no guarantee that someone would ever use this queue... IMHO listener should create a queue and publishers should not care about it at all. What do you think? On 06/27/2014 05:16 PM, Sandy Walsh wrote: Something to consider is the "create the queue in advance" feature is done for notifications, so we don't drop important messages on the floor by having an Exchange with no associated Queue. For RPC operations, this may not be required (we assume the service is available). If this check is truly a time-sink we could ignore that check for rpc calls. -S On 6/10/2014 9:31 AM, Alexei Kornienko wrote: Hi, Please find some answers inline. Regards, Alexei On 06/10/2014 03:06 PM, Flavio Percoco wrote: On 10/06/14 15:03 +0400, Dina Belova wrote: Hello, stackers! Oslo.messaging is future of how different OpenStack components communicate with each other, and really I'd love to start discussion about how we can make this library even better then it's now and how can we refactor it make more production-ready. As we all remember, oslo.messaging was initially inspired to be created as a logical continuation of nova.rpc - as a separated library, with lots of transports supported, etc. That's why oslo.messaging inherited not only advantages of now did the nova.rpc work (and it were lots of them), but also some architectural decisions that currently sometimes lead to the performance issues (we met some of them while Ceilometer performance testing [1] during the Icehouse). For instance, simple testing messaging server (with connection pool and eventlet) can process 700 messages per second. The same functionality implemented using plain kombu (without connection pool and eventlet) driver is processing ten times more - 7000-8000 messages per second. So we have the following suggestions about how we may make this process better and quicker (and really I'd love to collect your feedback, folks): 1) Currently we have main loop running in the Executor class, and I guess it'll be much better to move it to the Server class, as it'll make relationship between the classes easier and will leave Executor only one task - process the message and that's it (in blocking or eventlet mode). Moreover, this will make further refactoring much easier. To some extent, the executors are part of the server class since the later is the one actually controlling them. If I understood your proposal, the server class would implement the event loop, which means we would have an EventletServer / BlockingServer, right? If what I said is what you meant, then I disagree. Executors keep the eventloop isolated from other parts of the library and this is really important for us. One of the reason is to easily support multiple python versions - by having different event loops. Is my assumption correct? Could you elaborate more? No It's not how we plan it. Server will do the loop and pass received message to dispatcher and executor. It means that we would still have blocking executor and eventlet executor in the same server class. We would just change the implementation part to make it more consistent and easier to control. 2) Some of the drivers implementations (such as impl_rabbit and impl_qpid, for instance) are full of useless separated classes that in reality might be included to other ones. There are already some changes making the whole structure easier [2], and after the 1st issue will be solved Dispatcher and Listener also will be able to be refactored. This was done on purpose. The idea was to focus on backwards compatibility rather than cleaning up/improving the drivers. That said, sounds like those drivers could user some clean up. However, I think we should first extend the test suite a bit more before hacking the existing drivers. 3) If we'll separate RPC functionality and messaging functionality it'll make code base clean and easily reused. What do you mean with this? We mean that current drivers are written with RPC code hardcoded inside (ReplyWaiter, etc.). Thats not how messaging library is supposed to work. We can move RPC to a separate layer and this would be beneficial for both rpc (code will become more clean and less error-prone) and core messaging part (we'll be able to implement messaging in way that will work much faster). 4) Connection pool can be refactored to implement more efficient connection reusage. Please, elaborate. What changes do you envision? Currently there is a class that is called ConnectionContext that is used to manage pool. Additionaly it can be accessed/configured i
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Something to consider is the "create the queue in advance" feature is done for notifications, so we don't drop important messages on the floor by having an Exchange with no associated Queue. For RPC operations, this may not be required (we assume the service is available). If this check is truly a time-sink we could ignore that check for rpc calls. -S On 6/10/2014 9:31 AM, Alexei Kornienko wrote: Hi, Please find some answers inline. Regards, Alexei On 06/10/2014 03:06 PM, Flavio Percoco wrote: On 10/06/14 15:03 +0400, Dina Belova wrote: Hello, stackers! Oslo.messaging is future of how different OpenStack components communicate with each other, and really I’d love to start discussion about how we can make this library even better then it’s now and how can we refactor it make more production-ready. As we all remember, oslo.messaging was initially inspired to be created as a logical continuation of nova.rpc - as a separated library, with lots of transports supported, etc. That’s why oslo.messaging inherited not only advantages of now did the nova.rpc work (and it were lots of them), but also some architectural decisions that currently sometimes lead to the performance issues (we met some of them while Ceilometer performance testing [1] during the Icehouse). For instance, simple testing messaging server (with connection pool and eventlet) can process 700 messages per second. The same functionality implemented using plain kombu (without connection pool and eventlet) driver is processing ten times more - 7000-8000 messages per second. So we have the following suggestions about how we may make this process better and quicker (and really I’d love to collect your feedback, folks): 1) Currently we have main loop running in the Executor class, and I guess it’ll be much better to move it to the Server class, as it’ll make relationship between the classes easier and will leave Executor only one task - process the message and that’s it (in blocking or eventlet mode). Moreover, this will make further refactoring much easier. To some extent, the executors are part of the server class since the later is the one actually controlling them. If I understood your proposal, the server class would implement the event loop, which means we would have an EventletServer / BlockingServer, right? If what I said is what you meant, then I disagree. Executors keep the eventloop isolated from other parts of the library and this is really important for us. One of the reason is to easily support multiple python versions - by having different event loops. Is my assumption correct? Could you elaborate more? No It's not how we plan it. Server will do the loop and pass received message to dispatcher and executor. It means that we would still have blocking executor and eventlet executor in the same server class. We would just change the implementation part to make it more consistent and easier to control. 2) Some of the drivers implementations (such as impl_rabbit and impl_qpid, for instance) are full of useless separated classes that in reality might be included to other ones. There are already some changes making the whole structure easier [2], and after the 1st issue will be solved Dispatcher and Listener also will be able to be refactored. This was done on purpose. The idea was to focus on backwards compatibility rather than cleaning up/improving the drivers. That said, sounds like those drivers could user some clean up. However, I think we should first extend the test suite a bit more before hacking the existing drivers. 3) If we’ll separate RPC functionality and messaging functionality it’ll make code base clean and easily reused. What do you mean with this? We mean that current drivers are written with RPC code hardcoded inside (ReplyWaiter, etc.). Thats not how messaging library is supposed to work. We can move RPC to a separate layer and this would be beneficial for both rpc (code will become more clean and less error-prone) and core messaging part (we'll be able to implement messaging in way that will work much faster). 4) Connection pool can be refactored to implement more efficient connection reusage. Please, elaborate. What changes do you envision? Currently there is a class that is called ConnectionContext that is used to manage pool. Additionaly it can be accessed/configured in several other places. If we refactor it a little bit it would be much easier to use connections from the pool. As Dims suggested, I think filing some specs for this (and keeping the proposals separate) would help a lot in understanding what the exact plan is. Glad to know you're looking forward to help improving oslo.messaging. Thanks, Flavio Folks, are you ok with such a plan? Alexey Kornienko already started some of this work [2], but really we want to be sure that we chose the correct vector of development here. Thanks! [1] https://docs.google.com/document/d/ 1ARpKiYW2WN94JloG0prNcLjMeom-ySVhe8fvjXG_uR
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Hello, Please find some answers inline. Regards, Alexei On 06/27/2014 04:04 PM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 26/06/14 22:38, Alexei Kornienko wrote: Hello Jay, Benchmark for oslo.messaging is really simple: You create a client that sends messages infinitively and a server that processes them. After you can use rabbitmq management plugin to see average throughput in the queue. Simple example can be found here - https://github.com/andreykurilin/profiler-for-oslo-messaging I've mentioned some of this already in my previous mails but here it is again: "Huge" is descriptive but not quantitative :) Do you have any numbers that pinpoint the amount of time that is being spent reconstructing and declaring the queues, say, compared to the time spent doing transmission? I don't have precise slowdown percentage for each issue. I've just identified hotspots using cProfile and strace. This ten times number... is that an estimate or do you have hard numbers that show that? Just curious. Using a script that I've mentioned earlier I get average throughput on my PC ~700 cast calls per second. I've written a simple and stupid script that uses kombu directly (in a single threaded and synchronous way with single connection) It gave me throughput ~8000 messages per second. Thats why I say that library should work at least 10 times faster. It doesn't show that those major issues you've pointed out result in such large message processing speed dropdown though. Maybe there are other causes of slowdown we observe. Neither it shows that refactoring of the code will actually help to boost the library significantly. It doesn't show that those major issues are the *only* reason for slowdown. Bit it shows that those issues are biggest that are currently visible. We'll get a major speed boost if we fix them (and possibly discover new issues the would prevent us of reaching full speed). Though having some hard numbers from using kombu directly is still a good thing. Is it possible that we introduce some performance tests into oslo.messaging itself? Some performance tests may be introduced but they would be more like functional tests since they require setup of actual messaging server (rabbit, etc.). What do you mean exactly by performance tests? Just testing overall throughput with some basic scenarios or you mean finding hotspots in the code? Regards, Alexei Kornienko On 06/26/2014 11:22 PM, Jay Pipes wrote: Hey Alexei, thanks for sharing your findings. Comments inline. On 06/26/2014 10:08 AM, Alexei Kornienko wrote: Hello, Returning to performance issues of oslo.messaging. I've found 2 biggest issue in existing implementation of rabbit (kombu) driver: 1) For almost every message sent/received a new object of Consumer/Publisher class is created. And each object of this class tries to declare it's queue even if it's already declared. https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L159 This causes a huge slowdown. "Huge" is descriptive but not quantitative :) Do you have any numbers that pinpoint the amount of time that is being spent reconstructing and declaring the queues, say, compared to the time spent doing transmission? 2) with issue #1 is fixed (I've applied a small hack to fix it in my repo) the next big issue araise. For every rpc message received a reply is sent when processing is done (it seems that reply is sent even for "cast" calls which it really strange to me). Reply sent is using connection pool to "speed up" replies. Due to bad implementation of custom connection pool for every message sent underlying connection channel is closed and reopened: https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L689 Cause of this major issues oslo.messaging performance is at least 10 times slower than it could be. This ten times number... is that an estimate or do you have hard numbers that show that? Just curious. My opinion is that there is no simple and elegant fix for this issues in current implementation of oslo.messaging (most because of bad architecture of the library and processing flow). My proposal is that we should start working on new major release of messaging library with architectural issues fixed. This will allow us to avoid mentioned issues and provide much more performance and flexibility for end users. Main goal that we should achieve is separate rpc code from messaging code this will allow us to implement both parts in much simpler and cleaner way and in the same time it would be much faster. Perhaps actually a better starting point would be to create a benchmarking harness that will allow us to see some baseline throughput numbers that we can then compare to the iterative improvements you will push? Best, -jay Please share your thoughts on this topic. Regards, Alexei Kornienko On 06/16/2014 02:47 PM, Gordon Sim wrote: On 06/1
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 26/06/14 22:38, Alexei Kornienko wrote: > Hello Jay, > > Benchmark for oslo.messaging is really simple: You create a client > that sends messages infinitively and a server that processes them. > After you can use rabbitmq management plugin to see average > throughput in the queue. Simple example can be found here - > https://github.com/andreykurilin/profiler-for-oslo-messaging > > I've mentioned some of this already in my previous mails but here > it is again: > >> "Huge" is descriptive but not quantitative :) Do you have any >> numbers that pinpoint the amount of time that is being spent >> reconstructing and declaring the queues, say, compared to the >> time spent doing transmission? > I don't have precise slowdown percentage for each issue. I've just > identified hotspots using cProfile and strace. > >> This ten times number... is that an estimate or do you have hard >> numbers that show that? Just curious. > Using a script that I've mentioned earlier I get average throughput > on my PC ~700 cast calls per second. I've written a simple and > stupid script that uses kombu directly (in a single threaded and > synchronous way with single connection) It gave me throughput ~8000 > messages per second. Thats why I say that library should work at > least 10 times faster. It doesn't show that those major issues you've pointed out result in such large message processing speed dropdown though. Maybe there are other causes of slowdown we observe. Neither it shows that refactoring of the code will actually help to boost the library significantly. Though having some hard numbers from using kombu directly is still a good thing. Is it possible that we introduce some performance tests into oslo.messaging itself? > > Regards, Alexei Kornienko > > > On 06/26/2014 11:22 PM, Jay Pipes wrote: >> Hey Alexei, thanks for sharing your findings. Comments inline. >> >> On 06/26/2014 10:08 AM, Alexei Kornienko wrote: >>> Hello, >>> >>> Returning to performance issues of oslo.messaging. I've found 2 >>> biggest issue in existing implementation of rabbit (kombu) >>> driver: >>> >>> 1) For almost every message sent/received a new object of >>> Consumer/Publisher class is created. And each object of this >>> class tries to declare it's queue even if it's already >>> declared. >>> https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L159 >>> >>> >>> >>> This causes a huge slowdown. >> >> "Huge" is descriptive but not quantitative :) Do you have any >> numbers that pinpoint the amount of time that is being spent >> reconstructing and declaring the queues, say, compared to the >> time spent doing transmission? >> >>> 2) with issue #1 is fixed (I've applied a small hack to fix it >>> in my repo) the next big issue araise. For every rpc message >>> received a reply is sent when processing is done (it seems that >>> reply is sent even for "cast" calls which it really strange to >>> me). Reply sent is using connection pool to "speed up" replies. >>> Due to bad implementation of custom connection pool for every >>> message sent underlying connection channel is closed and >>> reopened: >>> https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L689 >>> >>> >>> >>> Cause of this major issues oslo.messaging performance is at least 10 >>> times slower than it could be. >> >> This ten times number... is that an estimate or do you have hard >> numbers that show that? Just curious. >> >>> My opinion is that there is no simple and elegant fix for this >>> issues in current implementation of oslo.messaging (most >>> because of bad architecture of the library and processing >>> flow). My proposal is that we should start working on new major >>> release of messaging library with architectural issues fixed. >>> This will allow us to avoid mentioned issues and provide much >>> more performance and flexibility for end users. Main goal that >>> we should achieve is separate rpc code from messaging code this >>> will allow us to implement both parts in much simpler and >>> cleaner way and in the same time it would be much faster. >> >> Perhaps actually a better starting point would be to create a >> benchmarking harness that will allow us to see some baseline >> throughput numbers that we can then compare to the iterative >> improvements you will push? >> >> Best, -jay >> >>> Please share your thoughts on this topic. >>> >>> Regards, Alexei Kornienko >>> >>> On 06/16/2014 02:47 PM, Gordon Sim wrote: On 06/13/2014 02:06 PM, Ihar Hrachyshka wrote: > On 10/06/14 15:40, Alexei Kornienko wrote: >> On 06/10/2014 03:59 PM, Gordon Sim wrote: >>> >>> I think there could be a lot of work required to >>> significantly improve that driver, and I wonder if that >>> would be better spent on e.g. the AMQP 1.0 driver which >>> I believe will perform much better and will off
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Hello Jay, Benchmark for oslo.messaging is really simple: You create a client that sends messages infinitively and a server that processes them. After you can use rabbitmq management plugin to see average throughput in the queue. Simple example can be found here - https://github.com/andreykurilin/profiler-for-oslo-messaging I've mentioned some of this already in my previous mails but here it is again: "Huge" is descriptive but not quantitative :) Do you have any numbers that pinpoint the amount of time that is being spent reconstructing and declaring the queues, say, compared to the time spent doing transmission? I don't have precise slowdown percentage for each issue. I've just identified hotspots using cProfile and strace. This ten times number... is that an estimate or do you have hard numbers that show that? Just curious. Using a script that I've mentioned earlier I get average throughput on my PC ~700 cast calls per second. I've written a simple and stupid script that uses kombu directly (in a single threaded and synchronous way with single connection) It gave me throughput ~8000 messages per second. Thats why I say that library should work at least 10 times faster. Regards, Alexei Kornienko On 06/26/2014 11:22 PM, Jay Pipes wrote: Hey Alexei, thanks for sharing your findings. Comments inline. On 06/26/2014 10:08 AM, Alexei Kornienko wrote: Hello, Returning to performance issues of oslo.messaging. I've found 2 biggest issue in existing implementation of rabbit (kombu) driver: 1) For almost every message sent/received a new object of Consumer/Publisher class is created. And each object of this class tries to declare it's queue even if it's already declared. https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L159 This causes a huge slowdown. "Huge" is descriptive but not quantitative :) Do you have any numbers that pinpoint the amount of time that is being spent reconstructing and declaring the queues, say, compared to the time spent doing transmission? 2) with issue #1 is fixed (I've applied a small hack to fix it in my repo) the next big issue araise. For every rpc message received a reply is sent when processing is done (it seems that reply is sent even for "cast" calls which it really strange to me). Reply sent is using connection pool to "speed up" replies. Due to bad implementation of custom connection pool for every message sent underlying connection channel is closed and reopened: https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L689 Cause of this major issues oslo.messaging performance is at least 10 times slower than it could be. This ten times number... is that an estimate or do you have hard numbers that show that? Just curious. My opinion is that there is no simple and elegant fix for this issues in current implementation of oslo.messaging (most because of bad architecture of the library and processing flow). My proposal is that we should start working on new major release of messaging library with architectural issues fixed. This will allow us to avoid mentioned issues and provide much more performance and flexibility for end users. Main goal that we should achieve is separate rpc code from messaging code this will allow us to implement both parts in much simpler and cleaner way and in the same time it would be much faster. Perhaps actually a better starting point would be to create a benchmarking harness that will allow us to see some baseline throughput numbers that we can then compare to the iterative improvements you will push? Best, -jay Please share your thoughts on this topic. Regards, Alexei Kornienko On 06/16/2014 02:47 PM, Gordon Sim wrote: On 06/13/2014 02:06 PM, Ihar Hrachyshka wrote: On 10/06/14 15:40, Alexei Kornienko wrote: On 06/10/2014 03:59 PM, Gordon Sim wrote: >>> I think there could be a lot of work required to significantly improve that driver, and I wonder if that would be better spent on e.g. the AMQP 1.0 driver which I believe will perform much better and will offer more choice in deployment. >> I agree with you on this. However I'm not sure that we can do such a decision. If we focus on amqp driver only we should mention it explicitly and deprecate qpid driver completely. There is no point in keeping driver that is not really functional. The driver is functional. It may be not that efficient as alternatives, but that's not a valid reason to deprecate it. The question in my view is what the plan is for ongoing development. Will the driver get better over time, or is it likely to remain as is at best (or even deteriorate)? Choice is good, but every choice adds to the maintenance burden, in testing against regressions if nothing else. I think an explicit decision about the future is beneficial, whatever the decision may be. ___ OpenStack-dev mailing list
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Hey Alexei, thanks for sharing your findings. Comments inline. On 06/26/2014 10:08 AM, Alexei Kornienko wrote: Hello, Returning to performance issues of oslo.messaging. I've found 2 biggest issue in existing implementation of rabbit (kombu) driver: 1) For almost every message sent/received a new object of Consumer/Publisher class is created. And each object of this class tries to declare it's queue even if it's already declared. https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L159 This causes a huge slowdown. "Huge" is descriptive but not quantitative :) Do you have any numbers that pinpoint the amount of time that is being spent reconstructing and declaring the queues, say, compared to the time spent doing transmission? 2) with issue #1 is fixed (I've applied a small hack to fix it in my repo) the next big issue araise. For every rpc message received a reply is sent when processing is done (it seems that reply is sent even for "cast" calls which it really strange to me). Reply sent is using connection pool to "speed up" replies. Due to bad implementation of custom connection pool for every message sent underlying connection channel is closed and reopened: https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L689 Cause of this major issues oslo.messaging performance is at least 10 times slower than it could be. This ten times number... is that an estimate or do you have hard numbers that show that? Just curious. My opinion is that there is no simple and elegant fix for this issues in current implementation of oslo.messaging (most because of bad architecture of the library and processing flow). My proposal is that we should start working on new major release of messaging library with architectural issues fixed. This will allow us to avoid mentioned issues and provide much more performance and flexibility for end users. Main goal that we should achieve is separate rpc code from messaging code this will allow us to implement both parts in much simpler and cleaner way and in the same time it would be much faster. Perhaps actually a better starting point would be to create a benchmarking harness that will allow us to see some baseline throughput numbers that we can then compare to the iterative improvements you will push? Best, -jay Please share your thoughts on this topic. Regards, Alexei Kornienko On 06/16/2014 02:47 PM, Gordon Sim wrote: On 06/13/2014 02:06 PM, Ihar Hrachyshka wrote: On 10/06/14 15:40, Alexei Kornienko wrote: On 06/10/2014 03:59 PM, Gordon Sim wrote: >>> I think there could be a lot of work required to significantly improve that driver, and I wonder if that would be better spent on e.g. the AMQP 1.0 driver which I believe will perform much better and will offer more choice in deployment. >> I agree with you on this. However I'm not sure that we can do such a decision. If we focus on amqp driver only we should mention it explicitly and deprecate qpid driver completely. There is no point in keeping driver that is not really functional. The driver is functional. It may be not that efficient as alternatives, but that's not a valid reason to deprecate it. The question in my view is what the plan is for ongoing development. Will the driver get better over time, or is it likely to remain as is at best (or even deteriorate)? Choice is good, but every choice adds to the maintenance burden, in testing against regressions if nothing else. I think an explicit decision about the future is beneficial, whatever the decision may be. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Hello, Returning to performance issues of oslo.messaging. I've found 2 biggest issue in existing implementation of rabbit (kombu) driver: 1) For almost every message sent/received a new object of Consumer/Publisher class is created. And each object of this class tries to declare it's queue even if it's already declared. https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L159 This causes a huge slowdown. 2) with issue #1 is fixed (I've applied a small hack to fix it in my repo) the next big issue araise. For every rpc message received a reply is sent when processing is done (it seems that reply is sent even for "cast" calls which it really strange to me). Reply sent is using connection pool to "speed up" replies. Due to bad implementation of custom connection pool for every message sent underlying connection channel is closed and reopened: https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/_drivers/impl_rabbit.py#L689 Cause of this major issues oslo.messaging performance is at least 10 times slower than it could be. My opinion is that there is no simple and elegant fix for this issues in current implementation of oslo.messaging (most because of bad architecture of the library and processing flow). My proposal is that we should start working on new major release of messaging library with architectural issues fixed. This will allow us to avoid mentioned issues and provide much more performance and flexibility for end users. Main goal that we should achieve is separate rpc code from messaging code this will allow us to implement both parts in much simpler and cleaner way and in the same time it would be much faster. Please share your thoughts on this topic. Regards, Alexei Kornienko On 06/16/2014 02:47 PM, Gordon Sim wrote: On 06/13/2014 02:06 PM, Ihar Hrachyshka wrote: On 10/06/14 15:40, Alexei Kornienko wrote: On 06/10/2014 03:59 PM, Gordon Sim wrote: >>> I think there could be a lot of work required to significantly improve that driver, and I wonder if that would be better spent on e.g. the AMQP 1.0 driver which I believe will perform much better and will offer more choice in deployment. >> I agree with you on this. However I'm not sure that we can do such a decision. If we focus on amqp driver only we should mention it explicitly and deprecate qpid driver completely. There is no point in keeping driver that is not really functional. The driver is functional. It may be not that efficient as alternatives, but that's not a valid reason to deprecate it. The question in my view is what the plan is for ongoing development. Will the driver get better over time, or is it likely to remain as is at best (or even deteriorate)? Choice is good, but every choice adds to the maintenance burden, in testing against regressions if nothing else. I think an explicit decision about the future is beneficial, whatever the decision may be. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
On 06/13/2014 02:06 PM, Ihar Hrachyshka wrote: On 10/06/14 15:40, Alexei Kornienko wrote: On 06/10/2014 03:59 PM, Gordon Sim wrote: >>> I think there could be a lot of work required to significantly improve that driver, and I wonder if that would be better spent on e.g. the AMQP 1.0 driver which I believe will perform much better and will offer more choice in deployment. >> I agree with you on this. However I'm not sure that we can do such a decision. If we focus on amqp driver only we should mention it explicitly and deprecate qpid driver completely. There is no point in keeping driver that is not really functional. The driver is functional. It may be not that efficient as alternatives, but that's not a valid reason to deprecate it. The question in my view is what the plan is for ongoing development. Will the driver get better over time, or is it likely to remain as is at best (or even deteriorate)? Choice is good, but every choice adds to the maintenance burden, in testing against regressions if nothing else. I think an explicit decision about the future is beneficial, whatever the decision may be. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 10/06/14 15:40, Alexei Kornienko wrote: > On 06/10/2014 03:59 PM, Gordon Sim wrote: >> On 06/10/2014 12:03 PM, Dina Belova wrote: >>> Hello, stackers! >>> >>> >>> Oslo.messaging is future of how different OpenStack components >>> communicate with each other, and really I’d love to start >>> discussion about how we can make this library even better then >>> it’s now and how can we refactor it make more >>> production-ready. >>> >>> >>> As we all remember, oslo.messaging was initially inspired to be >>> created as a logical continuation of nova.rpc - as a separated >>> library, with lots of transports supported, etc. That’s why >>> oslo.messaging inherited not only advantages of now did the >>> nova.rpc work (and it were lots of them), but also some >>> architectural decisions that currently sometimes lead to the >>> performance issues (we met some of them while Ceilometer >>> performance testing [1] during the Icehouse). >>> >>> >>> For instance, simple testing messaging server (with connection >>> pool and eventlet) can process 700 messages per second. The >>> same functionality implemented using plain kombu (without >>> connection pool and eventlet) driver is processing ten times >>> more - 7000-8000 messages per second. >>> >>> >>> So we have the following suggestions about how we may make this >>> process better and quicker (and really I’d love to collect your >>> feedback, folks): >>> >>> >>> 1) Currently we have main loop running in the Executor class, >>> and I guess it’ll be much better to move it to the Server >>> class, as it’ll make relationship between the classes easier >>> and will leave Executor only one task - process the message and >>> that’s it (in blocking or eventlet mode). Moreover, this will >>> make further refactoring much easier. >>> >>> 2) Some of the drivers implementations (such as impl_rabbit >>> and impl_qpid, for instance) are full of useless separated >>> classes that in reality might be included to other ones. There >>> are already some changes making the whole structure easier [2], >>> and after the 1st issue will be solved Dispatcher and Listener >>> also will be able to be refactored. >>> >>> 3) If we’ll separate RPC functionality and messaging >>> functionality it’ll make code base clean and easily reused. >>> >>> 4) Connection pool can be refactored to implement more >>> efficient connection reusage. >>> >>> >>> Folks, are you ok with such a plan? Alexey Kornienko already >>> started some of this work [2], but really we want to be sure >>> that we chose the correct vector of development here. >> >> For the impl_qpid driver, I think there would need to be quite >> significant changes to make it efficient. At present there are >> several synchronous roundtrips for every RPC call made[1]. >> Notifications are not treated any differently than RPCs (and >> sending a call is no different to sending a cast). >> >> I agree the connection pooling is not efficient. For qpid at >> least it creates too many connections for no real benefit[2]. >> >> I think this may be a result of trying to fit the same >> high-level design to two entirely different underlying APIs. >> >> For me at least, this also makes it hard to spot issues by >> reading the code. The qpid specific 'unit' tests for >> oslo.messaging also fail for me everytime when an actual qpidd >> broker is running (I haven't yet got to the bottom of that). >> >> I'm personally not sure that the changes to impl_qpid you linked >> to have much impact on either efficiency or readability, safety >> of the code. > Indeed it was only to remove some of the unnecessary complexity of > the code. We'll see more improvement after we'll implement points > 1,2 from the original email (cause the will allow us to proceed to > further improvement) > >> I think there could be a lot of work required to significantly >> improve that driver, and I wonder if that would be better spent >> on e.g. the AMQP 1.0 driver which I believe will perform much >> better and will offer more choice in deployment. > I agree with you on this. However I'm not sure that we can do such > a decision. If we focus on amqp driver only we should mention it > explicitly and deprecate qpid driver completely. There is no point > in keeping driver that is not really functional. The driver is functional. It may be not that efficient as alternatives, but that's not a valid reason to deprecate it. >> >> --Gordon >> >> [1] For both the request and the response, the sender is created >> every time, which results in at least one roundtrip to the >> broker. Again, for both the request and the response, the message >> is then sent with a blocking send, meaning a further synchronous >> round trip for each. So for an RPC call, instead of just one >> roundtrip, there are at least four. >> >> [2] In my view, what matters more than per-connection throughput >> for olso.messaging, is the scalability of the
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
On Wed, Jun 11, 2014 at 4:08 PM, Mark McLoughlin wrote: > Hi, > > On Tue, 2014-06-10 at 15:47 +0400, Dina Belova wrote: >> Dims, >> >> >> No problem with creating the specs, we just want to understand if the >> community is OK with our suggestions in general :) >> If so, I'll create the appropriate specs and we'll discuss them :) > > Personally, I find it difficult to understand the proposals as currently > described and how they address the performance problems you say you see. > > The specs process should help flesh out your ideas so they are more > understandable. On the other hand, it's pretty difficult to have an > abstract conversation about code re-factoring. So, some combination of > proof-of-concept patches and specs will probably work best. +1 I share Flavio's concern about the change to the server/executor relationship. Those classes were designed the way they are now after much discussion and thought, in order to let us support multiple threading models without each server having to know those details. Having a more detailed spec proposal to review will help clarify the sorts of changes being proposed. Doug > > Mark. > > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Hi, On Tue, 2014-06-10 at 15:47 +0400, Dina Belova wrote: > Dims, > > > No problem with creating the specs, we just want to understand if the > community is OK with our suggestions in general :) > If so, I'll create the appropriate specs and we'll discuss them :) Personally, I find it difficult to understand the proposals as currently described and how they address the performance problems you say you see. The specs process should help flesh out your ideas so they are more understandable. On the other hand, it's pretty difficult to have an abstract conversation about code re-factoring. So, some combination of proof-of-concept patches and specs will probably work best. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
On 06/10/2014 03:59 PM, Gordon Sim wrote: On 06/10/2014 12:03 PM, Dina Belova wrote: Hello, stackers! Oslo.messaging is future of how different OpenStack components communicate with each other, and really I’d love to start discussion about how we can make this library even better then it’s now and how can we refactor it make more production-ready. As we all remember, oslo.messaging was initially inspired to be created as a logical continuation of nova.rpc - as a separated library, with lots of transports supported, etc. That’s why oslo.messaging inherited not only advantages of now did the nova.rpc work (and it were lots of them), but also some architectural decisions that currently sometimes lead to the performance issues (we met some of them while Ceilometer performance testing [1] during the Icehouse). For instance, simple testing messaging server (with connection pool and eventlet) can process 700 messages per second. The same functionality implemented using plain kombu (without connection pool and eventlet) driver is processing ten times more - 7000-8000 messages per second. So we have the following suggestions about how we may make this process better and quicker (and really I’d love to collect your feedback, folks): 1) Currently we have main loop running in the Executor class, and I guess it’ll be much better to move it to the Server class, as it’ll make relationship between the classes easier and will leave Executor only one task - process the message and that’s it (in blocking or eventlet mode). Moreover, this will make further refactoring much easier. 2) Some of the drivers implementations (such as impl_rabbit and impl_qpid, for instance) are full of useless separated classes that in reality might be included to other ones. There are already some changes making the whole structure easier [2], and after the 1st issue will be solved Dispatcher and Listener also will be able to be refactored. 3) If we’ll separate RPC functionality and messaging functionality it’ll make code base clean and easily reused. 4) Connection pool can be refactored to implement more efficient connection reusage. Folks, are you ok with such a plan? Alexey Kornienko already started some of this work [2], but really we want to be sure that we chose the correct vector of development here. For the impl_qpid driver, I think there would need to be quite significant changes to make it efficient. At present there are several synchronous roundtrips for every RPC call made[1]. Notifications are not treated any differently than RPCs (and sending a call is no different to sending a cast). I agree the connection pooling is not efficient. For qpid at least it creates too many connections for no real benefit[2]. I think this may be a result of trying to fit the same high-level design to two entirely different underlying APIs. For me at least, this also makes it hard to spot issues by reading the code. The qpid specific 'unit' tests for oslo.messaging also fail for me everytime when an actual qpidd broker is running (I haven't yet got to the bottom of that). I'm personally not sure that the changes to impl_qpid you linked to have much impact on either efficiency or readability, safety of the code. Indeed it was only to remove some of the unnecessary complexity of the code. We'll see more improvement after we'll implement points 1,2 from the original email (cause the will allow us to proceed to further improvement) I think there could be a lot of work required to significantly improve that driver, and I wonder if that would be better spent on e.g. the AMQP 1.0 driver which I believe will perform much better and will offer more choice in deployment. I agree with you on this. However I'm not sure that we can do such a decision. If we focus on amqp driver only we should mention it explicitly and deprecate qpid driver completely. There is no point in keeping driver that is not really functional. --Gordon [1] For both the request and the response, the sender is created every time, which results in at least one roundtrip to the broker. Again, for both the request and the response, the message is then sent with a blocking send, meaning a further synchronous round trip for each. So for an RPC call, instead of just one roundtrip, there are at least four. [2] In my view, what matters more than per-connection throughput for olso.messaging, is the scalability of the system as you add many RPC clients and servers. Excessive creation of connections by each process will have a negative impact on this. I don't believe the current code gets anywhere close to the limits of the underlying connection and suspect it would be more efficient and faster to multiplex different streams down the same connection. This would be especially true where using eventlet I suspect. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.opensta
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
On 06/10/2014 12:03 PM, Dina Belova wrote: Hello, stackers! Oslo.messaging is future of how different OpenStack components communicate with each other, and really I’d love to start discussion about how we can make this library even better then it’s now and how can we refactor it make more production-ready. As we all remember, oslo.messaging was initially inspired to be created as a logical continuation of nova.rpc - as a separated library, with lots of transports supported, etc. That’s why oslo.messaging inherited not only advantages of now did the nova.rpc work (and it were lots of them), but also some architectural decisions that currently sometimes lead to the performance issues (we met some of them while Ceilometer performance testing [1] during the Icehouse). For instance, simple testing messaging server (with connection pool and eventlet) can process 700 messages per second. The same functionality implemented using plain kombu (without connection pool and eventlet) driver is processing ten times more - 7000-8000 messages per second. So we have the following suggestions about how we may make this process better and quicker (and really I’d love to collect your feedback, folks): 1) Currently we have main loop running in the Executor class, and I guess it’ll be much better to move it to the Server class, as it’ll make relationship between the classes easier and will leave Executor only one task - process the message and that’s it (in blocking or eventlet mode). Moreover, this will make further refactoring much easier. 2) Some of the drivers implementations (such as impl_rabbit and impl_qpid, for instance) are full of useless separated classes that in reality might be included to other ones. There are already some changes making the whole structure easier [2], and after the 1st issue will be solved Dispatcher and Listener also will be able to be refactored. 3) If we’ll separate RPC functionality and messaging functionality it’ll make code base clean and easily reused. 4) Connection pool can be refactored to implement more efficient connection reusage. Folks, are you ok with such a plan? Alexey Kornienko already started some of this work [2], but really we want to be sure that we chose the correct vector of development here. For the impl_qpid driver, I think there would need to be quite significant changes to make it efficient. At present there are several synchronous roundtrips for every RPC call made[1]. Notifications are not treated any differently than RPCs (and sending a call is no different to sending a cast). I agree the connection pooling is not efficient. For qpid at least it creates too many connections for no real benefit[2]. I think this may be a result of trying to fit the same high-level design to two entirely different underlying APIs. For me at least, this also makes it hard to spot issues by reading the code. The qpid specific 'unit' tests for oslo.messaging also fail for me everytime when an actual qpidd broker is running (I haven't yet got to the bottom of that). I'm personally not sure that the changes to impl_qpid you linked to have much impact on either efficiency or readability, safety of the code. I think there could be a lot of work required to significantly improve that driver, and I wonder if that would be better spent on e.g. the AMQP 1.0 driver which I believe will perform much better and will offer more choice in deployment. --Gordon [1] For both the request and the response, the sender is created every time, which results in at least one roundtrip to the broker. Again, for both the request and the response, the message is then sent with a blocking send, meaning a further synchronous round trip for each. So for an RPC call, instead of just one roundtrip, there are at least four. [2] In my view, what matters more than per-connection throughput for olso.messaging, is the scalability of the system as you add many RPC clients and servers. Excessive creation of connections by each process will have a negative impact on this. I don't believe the current code gets anywhere close to the limits of the underlying connection and suspect it would be more efficient and faster to multiplex different streams down the same connection. This would be especially true where using eventlet I suspect. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Hi, Please find some answers inline. Regards, Alexei On 06/10/2014 03:06 PM, Flavio Percoco wrote: On 10/06/14 15:03 +0400, Dina Belova wrote: Hello, stackers! Oslo.messaging is future of how different OpenStack components communicate with each other, and really I'd love to start discussion about how we can make this library even better then it's now and how can we refactor it make more production-ready. As we all remember, oslo.messaging was initially inspired to be created as a logical continuation of nova.rpc - as a separated library, with lots of transports supported, etc. That's why oslo.messaging inherited not only advantages of now did the nova.rpc work (and it were lots of them), but also some architectural decisions that currently sometimes lead to the performance issues (we met some of them while Ceilometer performance testing [1] during the Icehouse). For instance, simple testing messaging server (with connection pool and eventlet) can process 700 messages per second. The same functionality implemented using plain kombu (without connection pool and eventlet) driver is processing ten times more - 7000-8000 messages per second. So we have the following suggestions about how we may make this process better and quicker (and really I'd love to collect your feedback, folks): 1) Currently we have main loop running in the Executor class, and I guess it'll be much better to move it to the Server class, as it'll make relationship between the classes easier and will leave Executor only one task - process the message and that's it (in blocking or eventlet mode). Moreover, this will make further refactoring much easier. To some extent, the executors are part of the server class since the later is the one actually controlling them. If I understood your proposal, the server class would implement the event loop, which means we would have an EventletServer / BlockingServer, right? If what I said is what you meant, then I disagree. Executors keep the eventloop isolated from other parts of the library and this is really important for us. One of the reason is to easily support multiple python versions - by having different event loops. Is my assumption correct? Could you elaborate more? No It's not how we plan it. Server will do the loop and pass received message to dispatcher and executor. It means that we would still have blocking executor and eventlet executor in the same server class. We would just change the implementation part to make it more consistent and easier to control. 2) Some of the drivers implementations (such as impl_rabbit and impl_qpid, for instance) are full of useless separated classes that in reality might be included to other ones. There are already some changes making the whole structure easier [2], and after the 1st issue will be solved Dispatcher and Listener also will be able to be refactored. This was done on purpose. The idea was to focus on backwards compatibility rather than cleaning up/improving the drivers. That said, sounds like those drivers could user some clean up. However, I think we should first extend the test suite a bit more before hacking the existing drivers. 3) If we'll separate RPC functionality and messaging functionality it'll make code base clean and easily reused. What do you mean with this? We mean that current drivers are written with RPC code hardcoded inside (ReplyWaiter, etc.). Thats not how messaging library is supposed to work. We can move RPC to a separate layer and this would be beneficial for both rpc (code will become more clean and less error-prone) and core messaging part (we'll be able to implement messaging in way that will work much faster). 4) Connection pool can be refactored to implement more efficient connection reusage. Please, elaborate. What changes do you envision? Currently there is a class that is called ConnectionContext that is used to manage pool. Additionaly it can be accessed/configured in several other places. If we refactor it a little bit it would be much easier to use connections from the pool. As Dims suggested, I think filing some specs for this (and keeping the proposals separate) would help a lot in understanding what the exact plan is. Glad to know you're looking forward to help improving oslo.messaging. Thanks, Flavio Folks, are you ok with such a plan? Alexey Kornienko already started some of this work [2], but really we want to be sure that we chose the correct vector of development here. Thanks! [1] https://docs.google.com/document/d/ 1ARpKiYW2WN94JloG0prNcLjMeom-ySVhe8fvjXG_uRU/edit?usp=sharing [2] https://review.openstack.org/#/q/ status:open+owner:akornienko+project:openstack/oslo.messaging,n,z Best regards, Dina Belova Software Engineer Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
On 10/06/14 15:03 +0400, Dina Belova wrote: Hello, stackers! Oslo.messaging is future of how different OpenStack components communicate with each other, and really I’d love to start discussion about how we can make this library even better then it’s now and how can we refactor it make more production-ready. As we all remember, oslo.messaging was initially inspired to be created as a logical continuation of nova.rpc - as a separated library, with lots of transports supported, etc. That’s why oslo.messaging inherited not only advantages of now did the nova.rpc work (and it were lots of them), but also some architectural decisions that currently sometimes lead to the performance issues (we met some of them while Ceilometer performance testing [1] during the Icehouse). For instance, simple testing messaging server (with connection pool and eventlet) can process 700 messages per second. The same functionality implemented using plain kombu (without connection pool and eventlet) driver is processing ten times more - 7000-8000 messages per second. So we have the following suggestions about how we may make this process better and quicker (and really I’d love to collect your feedback, folks): 1) Currently we have main loop running in the Executor class, and I guess it’ll be much better to move it to the Server class, as it’ll make relationship between the classes easier and will leave Executor only one task - process the message and that’s it (in blocking or eventlet mode). Moreover, this will make further refactoring much easier. To some extent, the executors are part of the server class since the later is the one actually controlling them. If I understood your proposal, the server class would implement the event loop, which means we would have an EventletServer / BlockingServer, right? If what I said is what you meant, then I disagree. Executors keep the eventloop isolated from other parts of the library and this is really important for us. One of the reason is to easily support multiple python versions - by having different event loops. Is my assumption correct? Could you elaborate more? 2) Some of the drivers implementations (such as impl_rabbit and impl_qpid, for instance) are full of useless separated classes that in reality might be included to other ones. There are already some changes making the whole structure easier [2], and after the 1st issue will be solved Dispatcher and Listener also will be able to be refactored. This was done on purpose. The idea was to focus on backwards compatibility rather than cleaning up/improving the drivers. That said, sounds like those drivers could user some clean up. However, I think we should first extend the test suite a bit more before hacking the existing drivers. 3) If we’ll separate RPC functionality and messaging functionality it’ll make code base clean and easily reused. What do you mean with this? 4) Connection pool can be refactored to implement more efficient connection reusage. Please, elaborate. What changes do you envision? As Dims suggested, I think filing some specs for this (and keeping the proposals separate) would help a lot in understanding what the exact plan is. Glad to know you're looking forward to help improving oslo.messaging. Thanks, Flavio Folks, are you ok with such a plan? Alexey Kornienko already started some of this work [2], but really we want to be sure that we chose the correct vector of development here. Thanks! [1] https://docs.google.com/document/d/ 1ARpKiYW2WN94JloG0prNcLjMeom-ySVhe8fvjXG_uRU/edit?usp=sharing [2] https://review.openstack.org/#/q/ status:open+owner:akornienko+project:openstack/oslo.messaging,n,z Best regards, Dina Belova Software Engineer Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco pgpTe9lXrjjsY.pgp Description: PGP signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Dims, No problem with creating the specs, we just want to understand if the community is OK with our suggestions in general :) If so, I'll create the appropriate specs and we'll discuss them :) Thanks -- Dina On Tue, Jun 10, 2014 at 3:31 PM, Davanum Srinivas wrote: > Dina, Alexey, > > Do you mind filing some spec(s) please? > > http://markmail.org/message/yqhndsr3zrqcfwq4 > http://markmail.org/message/kpk35uikcnodq3jb > > thanks, > dims > > On Tue, Jun 10, 2014 at 7:03 AM, Dina Belova wrote: > > Hello, stackers! > > > > > > Oslo.messaging is future of how different OpenStack components > communicate > > with each other, and really I’d love to start discussion about how we can > > make this library even better then it’s now and how can we refactor it > make > > more production-ready. > > > > > > As we all remember, oslo.messaging was initially inspired to be created > as a > > logical continuation of nova.rpc - as a separated library, with lots of > > transports supported, etc. That’s why oslo.messaging inherited not only > > advantages of now did the nova.rpc work (and it were lots of them), but > also > > some architectural decisions that currently sometimes lead to the > > performance issues (we met some of them while Ceilometer performance > testing > > [1] during the Icehouse). > > > > > > For instance, simple testing messaging server (with connection pool and > > eventlet) can process 700 messages per second. The same functionality > > implemented using plain kombu (without connection pool and eventlet) > driver > > is processing ten times more - 7000-8000 messages per second. > > > > > > So we have the following suggestions about how we may make this process > > better and quicker (and really I’d love to collect your feedback, folks): > > > > > > 1) Currently we have main loop running in the Executor class, and I guess > > it’ll be much better to move it to the Server class, as it’ll make > > relationship between the classes easier and will leave Executor only one > > task - process the message and that’s it (in blocking or eventlet mode). > > Moreover, this will make further refactoring much easier. > > > > 2) Some of the drivers implementations (such as impl_rabbit and > impl_qpid, > > for instance) are full of useless separated classes that in reality > might be > > included to other ones. There are already some changes making the whole > > structure easier [2], and after the 1st issue will be solved Dispatcher > and > > Listener also will be able to be refactored. > > > > 3) If we’ll separate RPC functionality and messaging functionality it’ll > > make code base clean and easily reused. > > > > 4) Connection pool can be refactored to implement more efficient > connection > > reusage. > > > > > > Folks, are you ok with such a plan? Alexey Kornienko already started > some of > > this work [2], but really we want to be sure that we chose the correct > > vector of development here. > > > > > > Thanks! > > > > > > [1] > > > https://docs.google.com/document/d/1ARpKiYW2WN94JloG0prNcLjMeom-ySVhe8fvjXG_uRU/edit?usp=sharing > > > > [2] > > > https://review.openstack.org/#/q/status:open+owner:akornienko+project:openstack/oslo.messaging,n,z > > > > > > Best regards, > > > > Dina Belova > > > > Software Engineer > > > > Mirantis Inc. > > > > > > ___ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > -- > Davanum Srinivas :: http://davanum.wordpress.com > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Best regards, Dina Belova Software Engineer Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][messaging] Further improvements and refactoring
Dina, Alexey, Do you mind filing some spec(s) please? http://markmail.org/message/yqhndsr3zrqcfwq4 http://markmail.org/message/kpk35uikcnodq3jb thanks, dims On Tue, Jun 10, 2014 at 7:03 AM, Dina Belova wrote: > Hello, stackers! > > > Oslo.messaging is future of how different OpenStack components communicate > with each other, and really I’d love to start discussion about how we can > make this library even better then it’s now and how can we refactor it make > more production-ready. > > > As we all remember, oslo.messaging was initially inspired to be created as a > logical continuation of nova.rpc - as a separated library, with lots of > transports supported, etc. That’s why oslo.messaging inherited not only > advantages of now did the nova.rpc work (and it were lots of them), but also > some architectural decisions that currently sometimes lead to the > performance issues (we met some of them while Ceilometer performance testing > [1] during the Icehouse). > > > For instance, simple testing messaging server (with connection pool and > eventlet) can process 700 messages per second. The same functionality > implemented using plain kombu (without connection pool and eventlet) driver > is processing ten times more - 7000-8000 messages per second. > > > So we have the following suggestions about how we may make this process > better and quicker (and really I’d love to collect your feedback, folks): > > > 1) Currently we have main loop running in the Executor class, and I guess > it’ll be much better to move it to the Server class, as it’ll make > relationship between the classes easier and will leave Executor only one > task - process the message and that’s it (in blocking or eventlet mode). > Moreover, this will make further refactoring much easier. > > 2) Some of the drivers implementations (such as impl_rabbit and impl_qpid, > for instance) are full of useless separated classes that in reality might be > included to other ones. There are already some changes making the whole > structure easier [2], and after the 1st issue will be solved Dispatcher and > Listener also will be able to be refactored. > > 3) If we’ll separate RPC functionality and messaging functionality it’ll > make code base clean and easily reused. > > 4) Connection pool can be refactored to implement more efficient connection > reusage. > > > Folks, are you ok with such a plan? Alexey Kornienko already started some of > this work [2], but really we want to be sure that we chose the correct > vector of development here. > > > Thanks! > > > [1] > https://docs.google.com/document/d/1ARpKiYW2WN94JloG0prNcLjMeom-ySVhe8fvjXG_uRU/edit?usp=sharing > > [2] > https://review.openstack.org/#/q/status:open+owner:akornienko+project:openstack/oslo.messaging,n,z > > > Best regards, > > Dina Belova > > Software Engineer > > Mirantis Inc. > > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Davanum Srinivas :: http://davanum.wordpress.com ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo][messaging] Further improvements and refactoring
Hello, stackers! Oslo.messaging is future of how different OpenStack components communicate with each other, and really I’d love to start discussion about how we can make this library even better then it’s now and how can we refactor it make more production-ready. As we all remember, oslo.messaging was initially inspired to be created as a logical continuation of nova.rpc - as a separated library, with lots of transports supported, etc. That’s why oslo.messaging inherited not only advantages of now did the nova.rpc work (and it were lots of them), but also some architectural decisions that currently sometimes lead to the performance issues (we met some of them while Ceilometer performance testing [1] during the Icehouse). For instance, simple testing messaging server (with connection pool and eventlet) can process 700 messages per second. The same functionality implemented using plain kombu (without connection pool and eventlet) driver is processing ten times more - 7000-8000 messages per second. So we have the following suggestions about how we may make this process better and quicker (and really I’d love to collect your feedback, folks): 1) Currently we have main loop running in the Executor class, and I guess it’ll be much better to move it to the Server class, as it’ll make relationship between the classes easier and will leave Executor only one task - process the message and that’s it (in blocking or eventlet mode). Moreover, this will make further refactoring much easier. 2) Some of the drivers implementations (such as impl_rabbit and impl_qpid, for instance) are full of useless separated classes that in reality might be included to other ones. There are already some changes making the whole structure easier [2], and after the 1st issue will be solved Dispatcher and Listener also will be able to be refactored. 3) If we’ll separate RPC functionality and messaging functionality it’ll make code base clean and easily reused. 4) Connection pool can be refactored to implement more efficient connection reusage. Folks, are you ok with such a plan? Alexey Kornienko already started some of this work [2], but really we want to be sure that we chose the correct vector of development here. Thanks! [1] https://docs.google.com/document/d/1ARpKiYW2WN94JloG0prNcLjMeom-ySVhe8fvjXG_uRU/edit?usp=sharing [2] https://review.openstack.org/#/q/status:open+owner:akornienko+project:openstack/oslo.messaging,n,z Best regards, Dina Belova Software Engineer Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
