Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Wed, Jan 18 2017, Mehdi Abaakouk wrote: > So, I agree with gordc, perhaps you should stay with the old and > unsupported lib. And let other to use the supported one. The best option would be for the Monasca folks to actually participate upstream to the Kafka driver development and help

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

Thanks Joe for all these details, I can see that Monasca is still not able to switch to the new lib for new very good reasons. But according your comment on https://review.openstack.org/#/c/420579/ : I don't think that anyone currently using Monasca wants to accept either of those options so

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On 17/01/17 06:58 PM, Keen, Joe wrote: > > I also see additional problems with the use of the deprecated > SimpleConsumer and SimpleProducer clients. We really do need to > investigate migrating to the new async only Producer objects while still > maintaining the reliability guarantees that

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

Tony, I have some observations on the new client based on a short term test and a long running test. For short term use it uses 2x the memory compared to the older client. The logic that deals with receiving partial messages from Kafka was completely rewritten in the 1.x series and with logging

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

Mehdi, I'd support switching g-r to make oslo.messaging work. period. This is dragged on way too long. Thanks, Dims On Wed, Jan 11, 2017 at 2:25 AM, Mehdi Abaakouk wrote: > The library final release is really soon, and we are still blocked on > this topic. If this is not

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

The library final release is really soon, and we are still blocked on this topic. If this is not solved, we will release one more time an unusable driver in oslo.messging. I want to remember that people current uses the kafka driver in production with 'downstream patches' ready since 1 years to

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

Any progress ? On Thu, Dec 08, 2016 at 08:32:54AM +1100, Tony Breeds wrote: On Mon, Dec 05, 2016 at 04:03:13AM +, Keen, Joe wrote: I wasn’t able to set a test up on Friday and with all the other work I have for the next few days I doubt I’ll be able to get to it much before Wednesday.

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

Excerpts from Clint Byrum's message of 2016-12-02 09:44:40 -0800: > Excerpts from Tony Breeds's message of 2016-12-02 15:26:40 +1100: > > On Thu, Dec 01, 2016 at 08:41:54AM -0800, Joshua Harlow wrote: > > > Keen, Joe wrote: > > > > I¹ll look into testing the newest version of kafka-python and see

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Mon, Dec 05, 2016 at 04:03:13AM +, Keen, Joe wrote: > I don’t know, yet, that we can. Unless we can find an answer to the > questions I had above I’m not sure that this new library will be > performant and durable enough for the use cases Monasca has. I’m fairly > confident that we can

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Mon, Dec 05, 2016 at 04:03:13AM +, Keen, Joe wrote: > I wasn’t able to set a test up on Friday and with all the other work I > have for the next few days I doubt I’ll be able to get to it much before > Wednesday. It's Wednesday so can we have an update? Yours Tony. signature.asc

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On 12/4/16, 7:36 PM, "Tony Breeds" wrote: >On Fri, Dec 02, 2016 at 06:18:39PM +, Keen, Joe wrote: >> >> >> On 12/2/16, 1:29 AM, "Mehdi Abaakouk" wrote: >> >> >On Fri, Dec 02, 2016 at 03:29:59PM +1100, Tony Breeds wrote: >> >>On Thu, Dec 01,

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Fri, Dec 02, 2016 at 06:18:39PM +, Keen, Joe wrote: > > > On 12/2/16, 1:29 AM, "Mehdi Abaakouk" wrote: > > >On Fri, Dec 02, 2016 at 03:29:59PM +1100, Tony Breeds wrote: > >>On Thu, Dec 01, 2016 at 04:52:52PM +, Keen, Joe wrote: > >> > >>> Unfortunately there¹s

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On 12/2/16, 1:29 AM, "Mehdi Abaakouk" wrote: >On Fri, Dec 02, 2016 at 03:29:59PM +1100, Tony Breeds wrote: >>On Thu, Dec 01, 2016 at 04:52:52PM +, Keen, Joe wrote: >> >>> Unfortunately there¹s nothing wrong on the Monasca side so far as we >>>know. >>> We test new

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

Excerpts from Tony Breeds's message of 2016-12-02 15:26:40 +1100: > On Thu, Dec 01, 2016 at 08:41:54AM -0800, Joshua Harlow wrote: > > Keen, Joe wrote: > > > I¹ll look into testing the newest version of kafka-python and see if it > > > meets our needs. If it still isn¹t stable and performant

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Fri, Dec 02, 2016 at 09:39:41AM +0100, Mehdi Abaakouk wrote: On Fri, Dec 02, 2016 at 09:29:56AM +0100, Mehdi Abaakouk wrote: And my bench seems to confirm the perf issue have been solved: I have updated my requirement review to require >=1.3.1 [1] to solve the monasca issue. [1]

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Fri, Dec 02, 2016 at 09:29:56AM +0100, Mehdi Abaakouk wrote: And my bench seems to confirm the perf issue have been solved: I have updated my requirement review to require >=1.3.1 [1] to solve the monasca issue. [1] https://review.openstack.org/404878 -- Mehdi Abaakouk mail:

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Fri, Dec 02, 2016 at 03:29:59PM +1100, Tony Breeds wrote: On Thu, Dec 01, 2016 at 04:52:52PM +, Keen, Joe wrote: Unfortunately there’s nothing wrong on the Monasca side so far as we know. We test new versions of the kafka-python library outside of Monasca before we bother to try

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Fri, Dec 02, 2016 at 03:19:26PM +1100, Tony Breeds wrote: On Thu, Dec 01, 2016 at 07:57:37AM +0100, Mehdi Abaakouk wrote: I think the solution is pretty simple. Just Fix it. I'm not saying it's easy but it is *simple*. We're a group of skilled individuals We have several ways forward which

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Thu, Dec 01, 2016 at 04:52:52PM +, Keen, Joe wrote: > Unfortunately there’s nothing wrong on the Monasca side so far as we know. > We test new versions of the kafka-python library outside of Monasca > before we bother to try integrating a new version. Since 1.0 the > kafka-python library

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Thu, Dec 01, 2016 at 08:41:54AM -0800, Joshua Harlow wrote: > Keen, Joe wrote: > > I¹ll look into testing the newest version of kafka-python and see if it > > meets our needs. If it still isn¹t stable and performant enough what are > > the available options? > > Fix the kafka-python library

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Thu, Dec 01, 2016 at 07:57:37AM +0100, Mehdi Abaakouk wrote: > I'm aware of all of that, oslo.messaging patch for the new version is > ready since 8 months now. And we are still blocked... capping libraries, > whatever the reason, is very annoying and just freezes people work. And I agree with

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On 12/1/16, 9:44 AM, "Julien Danjou" wrote: >On Thu, Dec 01 2016, Keen, Joe wrote: > >Hi Joe, > >[…] > >> The message context wrapped around every message is extra overhead we >> don¹t want. >> There¹s no support for batch sends to Kafka. >> There¹s no support for keyed

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

Reopen them? If they weren't fixed, then seems like the right thing to try to do, and rinse and repeat until actually fixed :-P Keen, Joe wrote: > We reported > the bugs we found to the kafka-python project but they were closed once > they released a new version.

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

Damn, that's supposed to say co-installability... Lol, -Josh Joshua Harlow wrote: Keen, Joe wrote: I¹ll look into testing the newest version of kafka-python and see if it meets our needs. If it still isn¹t stable and performant enough what are the available options? Fix the kafka-python

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On 12/1/16, 9:41 AM, "Joshua Harlow" wrote: >Keen, Joe wrote: >> I¹ll look into testing the newest version of kafka-python and see if it >> meets our needs. If it still isn¹t stable and performant enough what >>are >> the available options? > >Fix the kafka-python

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Thu, Dec 01 2016, Keen, Joe wrote: Hi Joe, […] > The message context wrapped around every message is extra overhead we > don¹t want. > There¹s no support for batch sends to Kafka. > There¹s no support for keyed producers. > The forced auto_commit on consumers isn¹t something we can

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

Keen, Joe wrote: I¹ll look into testing the newest version of kafka-python and see if it meets our needs. If it still isn¹t stable and performant enough what are the available options? Fix the kafka-python library or fix monasca; those seem to be the options to me :) I'd also not like to

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On 12/1/16, 3:11 AM, "Julien Danjou" wrote: >On Thu, Dec 01 2016, Mehdi Abaakouk wrote: > >> I'm aware of all of that, oslo.messaging patch for the new version is >> ready since 8 months now. And we are still blocked... capping libraries, >> whatever the reason, is very

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Thu, Dec 01 2016, Mehdi Abaakouk wrote: > I'm aware of all of that, oslo.messaging patch for the new version is > ready since 8 months now. And we are still blocked... capping libraries, > whatever the reason, is very annoying and just freezes people work. > > From the API pov python-kafka

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

Le 2016-12-01 01:25, Tony Breeds a écrit : So right now I think the answer is still no we can't uncap it. To the best of my knowledge monasca still suffers a substatial performance decrease and the switch from sync to async producers isn't desired for them. Joe wrote this all up in:

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

On Wed, Nov 30, 2016 at 08:13:44PM +0100, Mehdi Abaakouk wrote: > > > Le 2016-11-30 19:11, Nadya Shakhat a écrit : > > Hi all, > > > > This one [1] is used very intensively actually :) It is not merged > > because > > we cannot just upgrade python-kafka version because of Monasca. We had > >