Re: [openstack-dev] [oslo][monasca][requirements] Kafka 1.x for Oslo.Messaging and Monasca
Tony, Psutil is the only library we’ve seen so far that was already in the global requirements with a version we didn’t support. Because of that we don’t currently have it as a requirement for the Monasca Agent, we wanted to use the upper constraints in our devstack gate jobs, and we just install it externally for now. Once that patch set lands we can add it back into the requirements for the Monasca Agent. We have several other libraries we’re in the process of removing that aren’t strictly necessary and that aren’t worth trying to add to global requirements right now. We currently have a review up to add our monasca-common repo to the requirements [1]. Once that goes through we should be able to immediately add reviews for our monasca-api and monasca-persister repos. I’m in the process of running some more Kafka tests and putting together more details about the Kafka problems from the Monasca perspective. The short version is that we have a simple Kafka test we used to validate version 0.9.5 that writes 100,000 messages to Kafka, at the average message size that Monasca expects, and then records how long it takes to consume them at a variety of fetch sizes from 50K to 2MB. With 0.9.5, depending on the fetch size, we could get throughput rates of 50K to 60K per second. With the 1.0.x series performance degraded down to 7K to 10K per second and after running my test program repeatedly it eventually consumed all the RAM in my VM and the kernel killed it. I recently tested version 1.2.2 and while the performance was better, a consistent ~40K per second regardless of fetch size, that only worked for a single run of my test program. A subsequent run only managed ~12K per second and did not complete the test. It consumed ~7GB of RAM before running my VM out of RAM. Aside from those performance issues the 1.x library is a fundamental change from the 0.9.x library. The producers became inherently asynchronous while Monasca requires synchronous writes. We should be able to configure the new producers to operate in a synchronous manner but we have not been able to verify that. The older synchronous producers have been deprecated and based on bugs filed seem to have degraded reliability. The consumers attempt to balance themselves inherently now if you’re using a Kafka version > 0.9 but we have not been able to verify that the new consumer code functions with the consumer balancing methods that are required with Kafka versions < 0.9. We do want to upgrade once things stabilize, especially since we want to take advantage of Kafka Streaming in the new 0.10 version of Kafka that the 0.9.5 library doesn’t support, but because we’re still running into performance and stability problems we haven’t been able to verify that the new versions can support the Monasca use case. [1] https://review.openstack.org/#/c/334625/ On 6/23/16, 9:37 PM, "Tony Breeds" wrote: >On Wed, Jun 22, 2016 at 05:34:27AM +, Keen, Joe wrote: >> Davanum, >> We started work on getting Monasca into the global requirements with >>two >> reviews [1] [2] that add gate jobs and check requirements jobs for the >> Monasca repositories. Some repositories are being adapted to use >>versions >> of libraries that OpenStack currently accepts [3] and we¹re looking at >>the >> libraries we use that are not currently part of OpenStack and seeing if >> they¹re worth trying to add to the global requirements. We¹re hoping to >> be able to start adding the global requirements reviews within a week or >> two. >> >> We definitely want to talk with the oslo.messaging team and explain the >> ways we use Kafka and what effects the move to the 1.x versions of the >> library has on us. I¹ve attempted to contact the oslo.messaging team in >> the oslo IRC channel to see if we can talk about this at a weekly >>meeting >> but I wasn¹t able to connect with anyone. Would you prefer that >> conversation happen on the mailing list here or could we add that topic >>to >> the next weekly meeting? >> >> [1] https://review.openstack.org/#/c/316293/ >> [2] https://review.openstack.org/#/c/323567/ > >These 2 are merged. > >> [3] https://review.openstack.org/#/c/323598/ > >Taking a tangent here: > >In 2014[1] we added a cap to psutil because 2.x wasn't compatible with 1.x >which is fine but 2 years latere we have 4.3.0 and because of the cap I'm >guessing we've done very little to work towards 4.3.0 > >I've added an item for the requirements team to look at what's involved in >raising the minimum for psutil, but: >Requirement: psutil<2.0.0,>=1.1.1 (used by 41 projects) >it wont happen soon. > >Is psutil the last of the "old" libraries you need to deal with? > >Getting back to the topic of kafka, what are the pain points involved in >working with the 2.x API? Clearly we're going to need to get monasca and >oslo.messgaing on a compatible page RSN or we'll end up delaying things >until >Ocata :( > >Yours Tony. > >[1] https://review.openstack.org/#/c/81
Re: [openstack-dev] [oslo][monasca][requirements] Kafka 1.x for Oslo.Messaging and Monasca
On Wed, Jun 22, 2016 at 05:34:27AM +, Keen, Joe wrote: > Davanum, > We started work on getting Monasca into the global requirements with two > reviews [1] [2] that add gate jobs and check requirements jobs for the > Monasca repositories. Some repositories are being adapted to use versions > of libraries that OpenStack currently accepts [3] and we¹re looking at the > libraries we use that are not currently part of OpenStack and seeing if > they¹re worth trying to add to the global requirements. We¹re hoping to > be able to start adding the global requirements reviews within a week or > two. > > We definitely want to talk with the oslo.messaging team and explain the > ways we use Kafka and what effects the move to the 1.x versions of the > library has on us. I¹ve attempted to contact the oslo.messaging team in > the oslo IRC channel to see if we can talk about this at a weekly meeting > but I wasn¹t able to connect with anyone. Would you prefer that > conversation happen on the mailing list here or could we add that topic to > the next weekly meeting? > > [1] https://review.openstack.org/#/c/316293/ > [2] https://review.openstack.org/#/c/323567/ These 2 are merged. > [3] https://review.openstack.org/#/c/323598/ Taking a tangent here: In 2014[1] we added a cap to psutil because 2.x wasn't compatible with 1.x which is fine but 2 years latere we have 4.3.0 and because of the cap I'm guessing we've done very little to work towards 4.3.0 I've added an item for the requirements team to look at what's involved in raising the minimum for psutil, but: Requirement: psutil<2.0.0,>=1.1.1 (used by 41 projects) it wont happen soon. Is psutil the last of the "old" libraries you need to deal with? Getting back to the topic of kafka, what are the pain points involved in working with the 2.x API? Clearly we're going to need to get monasca and oslo.messgaing on a compatible page RSN or we'll end up delaying things until Ocata :( Yours Tony. [1] https://review.openstack.org/#/c/81373/ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][monasca][requirements] Kafka 1.x for Oslo.Messaging and Monasca
Thanks for the update Joe. Let's start the discussion on the mailing list please. -- Dims On Wed, Jun 22, 2016 at 1:34 AM, Keen, Joe wrote: > Davanum, > We started work on getting Monasca into the global requirements with two > reviews [1] [2] that add gate jobs and check requirements jobs for the > Monasca repositories. Some repositories are being adapted to use versions > of libraries that OpenStack currently accepts [3] and we¹re looking at the > libraries we use that are not currently part of OpenStack and seeing if > they¹re worth trying to add to the global requirements. We¹re hoping to > be able to start adding the global requirements reviews within a week or > two. > > We definitely want to talk with the oslo.messaging team and explain the > ways we use Kafka and what effects the move to the 1.x versions of the > library has on us. I¹ve attempted to contact the oslo.messaging team in > the oslo IRC channel to see if we can talk about this at a weekly meeting > but I wasn¹t able to connect with anyone. Would you prefer that > conversation happen on the mailing list here or could we add that topic to > the next weekly meeting? > > [1] https://review.openstack.org/#/c/316293/ > [2] https://review.openstack.org/#/c/323567/ > [3] https://review.openstack.org/#/c/323598/ > > On 6/21/16, 6:53 AM, "Davanum Srinivas" wrote: > >>Roland Hochmuth, Joe Keen, >> >>The oslo.messaging folks have been trying off and on again [1] to get >>to python-kafka 1.x. Right now they are waiting on the Monasca team as >>we reverted python-kafka 1.x in [2] for the Monasca Team. >> >>I still don't see a review for adding Monasca to projects.txt which is >>concerning. Is there work being done to get to that point? >> >>The oslo.messaging team will file a review to update to 1.x again when >>they are ready, when that happens, we will end up breaking Monasca CI >>jobs again. Unless of course Monasca is in projects.txt by that time >>and adhering to requirements. >> >>If both oslo.messaging and Monasca are under requirements, we still >>need a way to move forward together. So please treat this email thread >>as an opportunity to talk to each other :) Which should have started >>ideally when we did [2] >> >>Thanks, >>Dims >> >>[1] >>https://review.openstack.org/#/q/kafka+project:openstack/oslo.messaging+NO >>T+is:merged+NOT+is:abandoned >>[2] https://review.openstack.org/#/c/316259/ >> >>-- >>Davanum Srinivas :: https://twitter.com/dims >> >>__ >>OpenStack Development Mailing List (not for usage questions) >>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Davanum Srinivas :: https://twitter.com/dims __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][monasca][requirements] Kafka 1.x for Oslo.Messaging and Monasca
Davanum, We started work on getting Monasca into the global requirements with two reviews [1] [2] that add gate jobs and check requirements jobs for the Monasca repositories. Some repositories are being adapted to use versions of libraries that OpenStack currently accepts [3] and we¹re looking at the libraries we use that are not currently part of OpenStack and seeing if they¹re worth trying to add to the global requirements. We¹re hoping to be able to start adding the global requirements reviews within a week or two. We definitely want to talk with the oslo.messaging team and explain the ways we use Kafka and what effects the move to the 1.x versions of the library has on us. I¹ve attempted to contact the oslo.messaging team in the oslo IRC channel to see if we can talk about this at a weekly meeting but I wasn¹t able to connect with anyone. Would you prefer that conversation happen on the mailing list here or could we add that topic to the next weekly meeting? [1] https://review.openstack.org/#/c/316293/ [2] https://review.openstack.org/#/c/323567/ [3] https://review.openstack.org/#/c/323598/ On 6/21/16, 6:53 AM, "Davanum Srinivas" wrote: >Roland Hochmuth, Joe Keen, > >The oslo.messaging folks have been trying off and on again [1] to get >to python-kafka 1.x. Right now they are waiting on the Monasca team as >we reverted python-kafka 1.x in [2] for the Monasca Team. > >I still don't see a review for adding Monasca to projects.txt which is >concerning. Is there work being done to get to that point? > >The oslo.messaging team will file a review to update to 1.x again when >they are ready, when that happens, we will end up breaking Monasca CI >jobs again. Unless of course Monasca is in projects.txt by that time >and adhering to requirements. > >If both oslo.messaging and Monasca are under requirements, we still >need a way to move forward together. So please treat this email thread >as an opportunity to talk to each other :) Which should have started >ideally when we did [2] > >Thanks, >Dims > >[1] >https://review.openstack.org/#/q/kafka+project:openstack/oslo.messaging+NO >T+is:merged+NOT+is:abandoned >[2] https://review.openstack.org/#/c/316259/ > >-- >Davanum Srinivas :: https://twitter.com/dims > >__ >OpenStack Development Mailing List (not for usage questions) >Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo][monasca][requirements] Kafka 1.x for Oslo.Messaging and Monasca
Roland Hochmuth, Joe Keen, The oslo.messaging folks have been trying off and on again [1] to get to python-kafka 1.x. Right now they are waiting on the Monasca team as we reverted python-kafka 1.x in [2] for the Monasca Team. I still don't see a review for adding Monasca to projects.txt which is concerning. Is there work being done to get to that point? The oslo.messaging team will file a review to update to 1.x again when they are ready, when that happens, we will end up breaking Monasca CI jobs again. Unless of course Monasca is in projects.txt by that time and adhering to requirements. If both oslo.messaging and Monasca are under requirements, we still need a way to move forward together. So please treat this email thread as an opportunity to talk to each other :) Which should have started ideally when we did [2] Thanks, Dims [1] https://review.openstack.org/#/q/kafka+project:openstack/oslo.messaging+NOT+is:merged+NOT+is:abandoned [2] https://review.openstack.org/#/c/316259/ -- Davanum Srinivas :: https://twitter.com/dims __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
