Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Julien Danjou]

On Wed, Jan 18 2017, Mehdi Abaakouk wrote:

> So, I agree with gordc, perhaps you should stay with the old and
> unsupported lib. And let other to use the supported one.

The best option would be for the Monasca folks to actually participate
upstream to the Kafka driver development and help getting the perf they
want. That would be more constructive than staying at old version and
finger-pointing the deficiencies of the new versions.

(Maybe they are doing that but I did not see any pointer toward that
direction).

-- 
Julien Danjou
;; Free Software hacker
;; https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Mehdi Abaakouk]

Thanks Joe for all these details, I can see that Monasca is still
not able to switch to the new lib for new very good reasons.

But according your comment on https://review.openstack.org/#/c/420579/ :


I don't think that anyone currently using Monasca wants to accept either
of those options so we need to find a way to maintain the current data
guarantees while using the async behaviour of the new client library.
That takes time and engineering effort to make that happen.  Is there
anyone in the community willing to put in the effort to help build and
test these new features at scale?


Nobody have plan to fix this issues soon.

And about from the same review:


On another topic I'm curious what new features are you looking to get
out of the new library.  Is there anything we can do to help you get the
capabilities you want with the existing client?


I don't think asking other projects to use deprecated and unsupported
lib version in new code is good, it's just adding fresh technical debt.

So, I agree with gordc, perhaps you should stay with the old and
unsupported lib. And let other to use the supported one.


On Tue, Jan 17, 2017 at 11:58:25PM +, Keen, Joe wrote:

Tony, I have some observations on the new client based on a short term
test and a long running test.

For short term use it uses 2x the memory compared to the older client.
The logic that deals with receiving partial messages from Kafka was
completely rewritten in the 1.x series and with logging enabled I see
continual warnings about truncated messages.  I don’t lose any data
because of this but I haven’t been able to verify if it’s doing more reads
than necessary.  I don’t know that either of these problems are really a
sticking point for Monasca but the increase in memory usage is potentially
a problem.



Long term testing showed some additional problems.  On a Kafka server that
has been running for a couple weeks I can write data in but the
kafka-python library is no longer able to read data from Kafka.  Clients
written in other languages are able to read successfully.  Profiling of
the python-kafka client shows that it’s spending all it’s time in a loop
attempting to connect to Kafka:

   276150.0860.0000.0860.000 {method 'acquire’ of
'thread.lock' objects}
431520.2500.0000.3850.000 types.py:15(_unpack)
431530.1350.0000.1350.000 {_struct.unpack}
48040/477980.1640.0000.1650.000 {len}
603510.2010.0000.2010.000 {method 'read’ of
'_io.BytesIO' objects}
  7389962   23.9850.000   23.9850.000 {method 'keys' of ‘dict'
objects}
  738  104.9310.000  395.6540.000 conn.py:560(recv)
  738   58.3420.000  100.0050.000
conn.py:722(_requests_timed_out)
  738   97.7870.000  167.5680.000 conn.py:588(_recv)
  7390071   46.5960.000   46.5960.000 {method 'recv’ of
'_socket.socket' objects}
  7390145   23.1510.000   23.1510.000 conn.py:458(connected)
  7390266   21.4170.000   21.4170.000 {method 'tell’ of
'_io.BytesIO' objects}
  7395664   41.6950.000   41.6950.000 {time.time}



I also see additional problems with the use of the deprecated
SimpleConsumer and SimpleProducer clients.  We really do need to
investigate migrating to the new async only Producer objects while still
maintaining the reliability guarantees that Monasca requires.


On 12/13/16, 10:01 PM, "Tony Breeds"  wrote:


On Mon, Dec 05, 2016 at 04:03:13AM +, Keen, Joe wrote:


I don’t know, yet, that we can.  Unless we can find an answer to the
questions I had above I’m not sure that this new library will be
performant and durable enough for the use cases Monasca has.  I’m fairly
confident that we can make it work but the performance issues with
previous versions prevented us from even trying to integrate so it will
take us some time.  If you need an answer more quickly than a week or
so,
and if anyone in the community is willing, I can walk them through the
testing I’d expect to happen to validate the new library.


Any updates Joe?  It's been 10 days and we're running close to Christamas
so
at this rate it'll be next year before we know if this is workable.

Yours Tony.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
Mehdi Abaakouk
mail: sil...@sileht.net
irc: sileht

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[gordon chung]

On 17/01/17 06:58 PM, Keen, Joe wrote:
>
> I also see additional problems with the use of the deprecated
> SimpleConsumer and SimpleProducer clients.  We really do need to
> investigate migrating to the new async only Producer objects while still
> maintaining the reliability guarantees that Monasca requires.
>

is there a reason why you are against bumping Kafka up for OpenStack? it 
seems Monasca requires 0.9.5 and is content with it. but the oslo team 
has developed something that works well for the use case for the broader 
OpenStack ecosystem.

this seems like a better solution would be to just allow monasca to stay 
as is and the openstack requirements to progress rather than get blocked 
by a service that may or may not be deployed.

the same thing was done when we bumped elasticsearch requirements. there 
are multiple projects using elasticsearch. we didn't have anyone working 
on it in Ceilometer so rather than block the entire community. we let it 
proceed and we could catch up later if it was urgent.

cheers,
-- 
gord
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Keen, Joe]

Tony, I have some observations on the new client based on a short term
test and a long running test.

For short term use it uses 2x the memory compared to the older client.
The logic that deals with receiving partial messages from Kafka was
completely rewritten in the 1.x series and with logging enabled I see
continual warnings about truncated messages.  I don’t lose any data
because of this but I haven’t been able to verify if it’s doing more reads
than necessary.  I don’t know that either of these problems are really a
sticking point for Monasca but the increase in memory usage is potentially
a problem.

Long term testing showed some additional problems.  On a Kafka server that
has been running for a couple weeks I can write data in but the
kafka-python library is no longer able to read data from Kafka.  Clients
written in other languages are able to read successfully.  Profiling of
the python-kafka client shows that it’s spending all it’s time in a loop
attempting to connect to Kafka:

276150.0860.0000.0860.000 {method 'acquire’ of
'thread.lock' objects}
431520.2500.0000.3850.000 types.py:15(_unpack)
431530.1350.0000.1350.000 {_struct.unpack}
48040/477980.1640.0000.1650.000 {len}
603510.2010.0000.2010.000 {method 'read’ of
'_io.BytesIO' objects}
  7389962   23.9850.000   23.9850.000 {method 'keys' of ‘dict'
objects}
  738  104.9310.000  395.6540.000 conn.py:560(recv)
  738   58.3420.000  100.0050.000
conn.py:722(_requests_timed_out)
  738   97.7870.000  167.5680.000 conn.py:588(_recv)
  7390071   46.5960.000   46.5960.000 {method 'recv’ of
'_socket.socket' objects}
  7390145   23.1510.000   23.1510.000 conn.py:458(connected)
  7390266   21.4170.000   21.4170.000 {method 'tell’ of
'_io.BytesIO' objects}
  7395664   41.6950.000   41.6950.000 {time.time}



I also see additional problems with the use of the deprecated
SimpleConsumer and SimpleProducer clients.  We really do need to
investigate migrating to the new async only Producer objects while still
maintaining the reliability guarantees that Monasca requires.


On 12/13/16, 10:01 PM, "Tony Breeds"  wrote:

>On Mon, Dec 05, 2016 at 04:03:13AM +, Keen, Joe wrote:
>
>> I don’t know, yet, that we can.  Unless we can find an answer to the
>> questions I had above I’m not sure that this new library will be
>> performant and durable enough for the use cases Monasca has.  I’m fairly
>> confident that we can make it work but the performance issues with
>> previous versions prevented us from even trying to integrate so it will
>> take us some time.  If you need an answer more quickly than a week or
>>so,
>> and if anyone in the community is willing, I can walk them through the
>> testing I’d expect to happen to validate the new library.
>
>Any updates Joe?  It's been 10 days and we're running close to Christamas
>so
>at this rate it'll be next year before we know if this is workable.
>
>Yours Tony.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Davanum Srinivas]

Mehdi,

I'd support switching g-r to make oslo.messaging work. period. This is
dragged on way too long.

Thanks,
Dims

On Wed, Jan 11, 2017 at 2:25 AM, Mehdi Abaakouk  wrote:
> The library final release is really soon, and we are still blocked on
> this topic. If this is not solved, we will release one more time an
> unusable driver in oslo.messging.
>
> I want to remember that people current uses the kafka driver in
> production with 'downstream patches' ready since 1 years to make it
> works.
>
> We recently remove the kafka dep from oslo.messaging to be able to merge
> some of these patches. But we can't untag the experimental flag of
> this driver until the dependency issue is solved.
>
> So what can we do to unblock this situation ?
>
>
> On Fri, Jan 06, 2017 at 02:31:28PM +0100, Mehdi Abaakouk wrote:
>>
>> Any progress ?
>>
>> On Thu, Dec 08, 2016 at 08:32:54AM +1100, Tony Breeds wrote:
>>>
>>> On Mon, Dec 05, 2016 at 04:03:13AM +, Keen, Joe wrote:

 I wasn’t able to set a test up on Friday and with all the other work I
 have for the next few days I doubt I’ll be able to get to it much before
 Wednesday.
>>>
>>>
>>> It's Wednesday so can we have an update?
>>>
>>> Yours Tony.
>>
>>
>> --
>> Mehdi Abaakouk
>> mail: sil...@sileht.net
>>
>> irc: sileht
>
>
> --
> Mehdi Abaakouk
> mail: sil...@sileht.net
> ]]irc: sileht
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Davanum Srinivas :: https://twitter.com/dims

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Mehdi Abaakouk]

The library final release is really soon, and we are still blocked on
this topic. If this is not solved, we will release one more time an
unusable driver in oslo.messging.

I want to remember that people current uses the kafka driver in
production with 'downstream patches' ready since 1 years to make it
works.

We recently remove the kafka dep from oslo.messaging to be able to merge
some of these patches. But we can't untag the experimental flag of
this driver until the dependency issue is solved.

So what can we do to unblock this situation ?

On Fri, Jan 06, 2017 at 02:31:28PM +0100, Mehdi Abaakouk wrote:

Any progress ?

On Thu, Dec 08, 2016 at 08:32:54AM +1100, Tony Breeds wrote:

On Mon, Dec 05, 2016 at 04:03:13AM +, Keen, Joe wrote:

I wasn’t able to set a test up on Friday and with all the other work I
have for the next few days I doubt I’ll be able to get to it much before
Wednesday.


It's Wednesday so can we have an update?

Yours Tony.


--
Mehdi Abaakouk
mail: sil...@sileht.net

irc: sileht


--
Mehdi Abaakouk
mail: sil...@sileht.net
]]irc: sileht

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Mehdi Abaakouk]

Any progress ?

On Thu, Dec 08, 2016 at 08:32:54AM +1100, Tony Breeds wrote:

On Mon, Dec 05, 2016 at 04:03:13AM +, Keen, Joe wrote:

I wasn’t able to set a test up on Friday and with all the other work I
have for the next few days I doubt I’ll be able to get to it much before
Wednesday.


It's Wednesday so can we have an update?

Yours Tony.


--
Mehdi Abaakouk
mail: sil...@sileht.net

irc: sileht

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Doug Hellmann]

Excerpts from Clint Byrum's message of 2016-12-02 09:44:40 -0800:
> Excerpts from Tony Breeds's message of 2016-12-02 15:26:40 +1100:
> > On Thu, Dec 01, 2016 at 08:41:54AM -0800, Joshua Harlow wrote:
> > > Keen, Joe wrote:
> > > > I¹ll look into testing the newest version of kafka-python and see if it
> > > > meets our needs.  If it still isn¹t stable and performant enough what 
> > > > are
> > > > the available options?
> > > 
> > > Fix the kafka-python library or fix monasca; those seem to be the options 
> > > to
> > > me :)
> > 
> > Yup, Also worth including fix oslo.messaging to meet monasca's needs.  But
> > *something* needs fixing.
> >  
> > > I'd also not like to block the rest of the world (from using newer 
> > > versions
> > > of kafka-python) during this as well. But then this may diverge/expand 
> > > into
> > > a discussion we had a few summits ago, about getting rid of
> > > co-instability...
> > 
> > lalalalala not listening ;P
> > 
> > Less flippantly, there are a couple of ways to do this but IMO they're not 
> > in
> > the best interest of OpenStack.
> > 
> > 1. vendor/fork python-kafka 0.X
> > 2. Stop the proposal-bot from syncing with monasca, thereby allowing it to 
> > use
> > python-kafka 0.X at the expense of co-installability.
> 
> Could there be a (3)?
> 
>  3. change the oslo driver to work with the currently pinned
>  python-kafka version?

Isn't the point that the pinned version, 0.9.5, is out of date? The
current version is 1.3.1.

Doug

> 
> > 
> > Fortunately either option is easy to reverse once the underlying issue is 
> > fixed.
> > 
> > Yours Tony.
> 

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Tony Breeds]

On Mon, Dec 05, 2016 at 04:03:13AM +, Keen, Joe wrote:

> I don’t know, yet, that we can.  Unless we can find an answer to the
> questions I had above I’m not sure that this new library will be
> performant and durable enough for the use cases Monasca has.  I’m fairly
> confident that we can make it work but the performance issues with
> previous versions prevented us from even trying to integrate so it will
> take us some time.  If you need an answer more quickly than a week or so,
> and if anyone in the community is willing, I can walk them through the
> testing I’d expect to happen to validate the new library.

Any updates Joe?  It's been 10 days and we're running close to Christamas so
at this rate it'll be next year before we know if this is workable.

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Tony Breeds]

On Mon, Dec 05, 2016 at 04:03:13AM +, Keen, Joe wrote:
> I wasn’t able to set a test up on Friday and with all the other work I
> have for the next few days I doubt I’ll be able to get to it much before
> Wednesday.

It's Wednesday so can we have an update?

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Keen, Joe]

On 12/4/16, 7:36 PM, "Tony Breeds"  wrote:

>On Fri, Dec 02, 2016 at 06:18:39PM +, Keen, Joe wrote:
>> 
>> 
>> On 12/2/16, 1:29 AM, "Mehdi Abaakouk"  wrote:
>> 
>> >On Fri, Dec 02, 2016 at 03:29:59PM +1100, Tony Breeds wrote:
>> >>On Thu, Dec 01, 2016 at 04:52:52PM +, Keen, Joe wrote:
>> >>
>> >>> Unfortunately there¹s nothing wrong on the Monasca side so far as we
>> >>>know.
>> >>>  We test new versions of the kafka-python library outside of Monasca
>> >>> before we bother to try integrating a new version.  Since 1.0 the
>> >>> kafka-python library has suffered from crashes and memory leaks
>>severe
>> >>> enough that we¹ve never attempted using it in Monasca itself.  We
>> >>>reported
>> >>> the bugs we found to the kafka-python project but they were closed
>>once
>> >>> they released a new version.
>> >>
>> >>So Opening bugs isn't working.  What about writing code?
>> >
>> >The bug https://github.com/dpkp/kafka-python/issues/55
>> >
>> >Reopening it would be the right solution here.
>> >
>> >I can't reproduce the segfault neither and I agree with dpkp, that
>>looks
>> >like a
>> >ujson issue.
>> 
>> 
>> The bug I had was: https://github.com/dpkp/kafka-python/issues/551
>> 
>> In the case of that bug ujson was not an issue.  The behaviour remained
>> even using the standard json library.  The primary issue I found with it
>> was a memory leak over successive runs of the test script.  Eventually
>>the
>> leak became so bad that the OOM killer killed the process which caused
>>the
>> segfault I was seeing.  The last version I tested was 1.2.1 and it still
>> leaked badly.  I¹ll need to let the benchmark script run for a while and
>> make sure it¹s not still leaking.
>
>So you write that on Friday so you shoudl know by now if it's leaking
>care to
>give us an update?

I wasn’t able to set a test up on Friday and with all the other work I
have for the next few days I doubt I’ll be able to get to it much before
Wednesday.

>
>> >And my bench seems to confirm the perf issue have been solved:
>> >(but not in the pointed version...)
>> >
>> >$ pifpaf run kafka python kafka_test.py
>> >kafka-python version: 0.9.5
>> >...
>> >fetch size 179200 -> 45681.8728864 messages per second
>> >fetch size 204800 -> 47724.3810674 messages per second
>> >fetch size 230400 -> 47209.9841092 messages per second
>> >fetch size 256000 -> 48340.7719787 messages per second
>> >fetch size 281600 -> 49192.9896743 messages per second
>> >fetch size 307200 -> 50915.3291133 messages per second
>> >
>> >$ pifpaf run kafka python kafka_test.py
>> >kafka-python version: 1.0.2
>> >
>> >fetch size 179200 -> 8546.77931323 messages per second
>> >fetch size 204800 -> 9213.30958314 messages per second
>> >fetch size 230400 -> 10316.668006 messages per second
>> >fetch size 256000 -> 11476.2285269 messages per second
>> >fetch size 281600 -> 12353.7254386 messages per second
>> >fetch size 307200 -> 13131.2367288 messages per second
>> >
>> >(1.1.1 and 1.2.5 have also the same issue)
>> >
>> >$ pifpaf run kafka python kafka_test.py
>> >kafka-python version: 1.3.1
>> >fetch size 179200 -> 44636.9371873 messages per second
>> >fetch size 204800 -> 44324.7085365 messages per second
>> >fetch size 230400 -> 45235.8283208 messages per second
>> >fetch size 256000 -> 45793.1044121 messages per second
>> >fetch size 281600 -> 44648.6357019 messages per second
>> >fetch size 307200 -> 44877.8445987 messages per second
>> >fetch size 332800 -> 47166.9176281 messages per second
>> >fetch size 358400 -> 47391.0057622 messages per second
>> >
>> >Looks like it works well now :)
>> 
>> It¹s good that the performance problem has been fixed.  The remaining
>> issues on the Monasca side are verifying that the batch send method we
>> were using in 0.9.5 still works with the new async behaviour, seeing if
>> our consumer auto balance still functions or converting to use the Kafka
>> internal auto balance in Kafka 0.10, and finding a way to do efficient
>> synchronous writes with the new async methods.
>
>Can you +1 https://review.openstack.org/404878 to make it clear we can
>proceed
>down this path.

I don’t know, yet, that we can.  Unless we can find an answer to the
questions I had above I’m not sure that this new library will be
performant and durable enough for the use cases Monasca has.  I’m fairly
confident that we can make it work but the performance issues with
previous versions prevented us from even trying to integrate so it will
take us some time.  If you need an answer more quickly than a week or so,
and if anyone in the community is willing, I can walk them through the
testing I’d expect to happen to validate the new library.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Tony Breeds]

On Fri, Dec 02, 2016 at 06:18:39PM +, Keen, Joe wrote:
> 
> 
> On 12/2/16, 1:29 AM, "Mehdi Abaakouk"  wrote:
> 
> >On Fri, Dec 02, 2016 at 03:29:59PM +1100, Tony Breeds wrote:
> >>On Thu, Dec 01, 2016 at 04:52:52PM +, Keen, Joe wrote:
> >>
> >>> Unfortunately there¹s nothing wrong on the Monasca side so far as we
> >>>know.
> >>>  We test new versions of the kafka-python library outside of Monasca
> >>> before we bother to try integrating a new version.  Since 1.0 the
> >>> kafka-python library has suffered from crashes and memory leaks severe
> >>> enough that we¹ve never attempted using it in Monasca itself.  We
> >>>reported
> >>> the bugs we found to the kafka-python project but they were closed once
> >>> they released a new version.
> >>
> >>So Opening bugs isn't working.  What about writing code?
> >
> >The bug https://github.com/dpkp/kafka-python/issues/55
> >
> >Reopening it would be the right solution here.
> >
> >I can't reproduce the segfault neither and I agree with dpkp, that looks
> >like a
> >ujson issue.
> 
> 
> The bug I had was: https://github.com/dpkp/kafka-python/issues/551
> 
> In the case of that bug ujson was not an issue.  The behaviour remained
> even using the standard json library.  The primary issue I found with it
> was a memory leak over successive runs of the test script.  Eventually the
> leak became so bad that the OOM killer killed the process which caused the
> segfault I was seeing.  The last version I tested was 1.2.1 and it still
> leaked badly.  I¹ll need to let the benchmark script run for a while and
> make sure it¹s not still leaking.

So you write that on Friday so you shoudl know by now if it's leaking care to
give us an update?

> >And my bench seems to confirm the perf issue have been solved:
> >(but not in the pointed version...)
> >
> >$ pifpaf run kafka python kafka_test.py
> >kafka-python version: 0.9.5
> >...
> >fetch size 179200 -> 45681.8728864 messages per second
> >fetch size 204800 -> 47724.3810674 messages per second
> >fetch size 230400 -> 47209.9841092 messages per second
> >fetch size 256000 -> 48340.7719787 messages per second
> >fetch size 281600 -> 49192.9896743 messages per second
> >fetch size 307200 -> 50915.3291133 messages per second
> >
> >$ pifpaf run kafka python kafka_test.py
> >kafka-python version: 1.0.2
> >
> >fetch size 179200 -> 8546.77931323 messages per second
> >fetch size 204800 -> 9213.30958314 messages per second
> >fetch size 230400 -> 10316.668006 messages per second
> >fetch size 256000 -> 11476.2285269 messages per second
> >fetch size 281600 -> 12353.7254386 messages per second
> >fetch size 307200 -> 13131.2367288 messages per second
> >
> >(1.1.1 and 1.2.5 have also the same issue)
> >
> >$ pifpaf run kafka python kafka_test.py
> >kafka-python version: 1.3.1
> >fetch size 179200 -> 44636.9371873 messages per second
> >fetch size 204800 -> 44324.7085365 messages per second
> >fetch size 230400 -> 45235.8283208 messages per second
> >fetch size 256000 -> 45793.1044121 messages per second
> >fetch size 281600 -> 44648.6357019 messages per second
> >fetch size 307200 -> 44877.8445987 messages per second
> >fetch size 332800 -> 47166.9176281 messages per second
> >fetch size 358400 -> 47391.0057622 messages per second
> >
> >Looks like it works well now :)
> 
> It¹s good that the performance problem has been fixed.  The remaining
> issues on the Monasca side are verifying that the batch send method we
> were using in 0.9.5 still works with the new async behaviour, seeing if
> our consumer auto balance still functions or converting to use the Kafka
> internal auto balance in Kafka 0.10, and finding a way to do efficient
> synchronous writes with the new async methods.

Can you +1 https://review.openstack.org/404878 to make it clear we can proceed
down this path.

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Keen, Joe]

On 12/2/16, 1:29 AM, "Mehdi Abaakouk"  wrote:

>On Fri, Dec 02, 2016 at 03:29:59PM +1100, Tony Breeds wrote:
>>On Thu, Dec 01, 2016 at 04:52:52PM +, Keen, Joe wrote:
>>
>>> Unfortunately there¹s nothing wrong on the Monasca side so far as we
>>>know.
>>>  We test new versions of the kafka-python library outside of Monasca
>>> before we bother to try integrating a new version.  Since 1.0 the
>>> kafka-python library has suffered from crashes and memory leaks severe
>>> enough that we¹ve never attempted using it in Monasca itself.  We
>>>reported
>>> the bugs we found to the kafka-python project but they were closed once
>>> they released a new version.
>>
>>So Opening bugs isn't working.  What about writing code?
>
>The bug https://github.com/dpkp/kafka-python/issues/55
>
>Reopening it would be the right solution here.
>
>I can't reproduce the segfault neither and I agree with dpkp, that looks
>like a
>ujson issue.


The bug I had was: https://github.com/dpkp/kafka-python/issues/551

In the case of that bug ujson was not an issue.  The behaviour remained
even using the standard json library.  The primary issue I found with it
was a memory leak over successive runs of the test script.  Eventually the
leak became so bad that the OOM killer killed the process which caused the
segfault I was seeing.  The last version I tested was 1.2.1 and it still
leaked badly.  I¹ll need to let the benchmark script run for a while and
make sure it¹s not still leaking.

>
>And my bench seems to confirm the perf issue have been solved:
>(but not in the pointed version...)
>
>$ pifpaf run kafka python kafka_test.py
>kafka-python version: 0.9.5
>...
>fetch size 179200 -> 45681.8728864 messages per second
>fetch size 204800 -> 47724.3810674 messages per second
>fetch size 230400 -> 47209.9841092 messages per second
>fetch size 256000 -> 48340.7719787 messages per second
>fetch size 281600 -> 49192.9896743 messages per second
>fetch size 307200 -> 50915.3291133 messages per second
>
>$ pifpaf run kafka python kafka_test.py
>kafka-python version: 1.0.2
>
>fetch size 179200 -> 8546.77931323 messages per second
>fetch size 204800 -> 9213.30958314 messages per second
>fetch size 230400 -> 10316.668006 messages per second
>fetch size 256000 -> 11476.2285269 messages per second
>fetch size 281600 -> 12353.7254386 messages per second
>fetch size 307200 -> 13131.2367288 messages per second
>
>(1.1.1 and 1.2.5 have also the same issue)
>
>$ pifpaf run kafka python kafka_test.py
>kafka-python version: 1.3.1
>fetch size 179200 -> 44636.9371873 messages per second
>fetch size 204800 -> 44324.7085365 messages per second
>fetch size 230400 -> 45235.8283208 messages per second
>fetch size 256000 -> 45793.1044121 messages per second
>fetch size 281600 -> 44648.6357019 messages per second
>fetch size 307200 -> 44877.8445987 messages per second
>fetch size 332800 -> 47166.9176281 messages per second
>fetch size 358400 -> 47391.0057622 messages per second
>
>Looks like it works well now :)

It¹s good that the performance problem has been fixed.  The remaining
issues on the Monasca side are verifying that the batch send method we
were using in 0.9.5 still works with the new async behaviour, seeing if
our consumer auto balance still functions or converting to use the Kafka
internal auto balance in Kafka 0.10, and finding a way to do efficient
synchronous writes with the new async methods.


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Clint Byrum]

Excerpts from Tony Breeds's message of 2016-12-02 15:26:40 +1100:
> On Thu, Dec 01, 2016 at 08:41:54AM -0800, Joshua Harlow wrote:
> > Keen, Joe wrote:
> > > I¹ll look into testing the newest version of kafka-python and see if it
> > > meets our needs.  If it still isn¹t stable and performant enough what are
> > > the available options?
> > 
> > Fix the kafka-python library or fix monasca; those seem to be the options to
> > me :)
> 
> Yup, Also worth including fix oslo.messaging to meet monasca's needs.  But
> *something* needs fixing.
>  
> > I'd also not like to block the rest of the world (from using newer versions
> > of kafka-python) during this as well. But then this may diverge/expand into
> > a discussion we had a few summits ago, about getting rid of
> > co-instability...
> 
> lalalalala not listening ;P
> 
> Less flippantly, there are a couple of ways to do this but IMO they're not in
> the best interest of OpenStack.
> 
> 1. vendor/fork python-kafka 0.X
> 2. Stop the proposal-bot from syncing with monasca, thereby allowing it to use
> python-kafka 0.X at the expense of co-installability.

Could there be a (3)?

 3. change the oslo driver to work with the currently pinned
 python-kafka version?

> 
> Fortunately either option is easy to reverse once the underlying issue is 
> fixed.
> 
> Yours Tony.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Mehdi Abaakouk]

On Fri, Dec 02, 2016 at 09:39:41AM +0100, Mehdi Abaakouk wrote:

On Fri, Dec 02, 2016 at 09:29:56AM +0100, Mehdi Abaakouk wrote:

And my bench seems to confirm the perf issue have been solved:


I have updated my requirement review to require >=1.3.1 [1] to solve
the monasca issue.

[1] https://review.openstack.org/404878a


And this is the update for all projects:

https://review.openstack.org/#/q/status:open+branch:master+topic:sileht/kafka-update

Nothing should block all of this anymore, except +2/+A :)

--
Mehdi Abaakouk
mail: sil...@sileht.net
irc: sileht

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Mehdi Abaakouk]

On Fri, Dec 02, 2016 at 09:29:56AM +0100, Mehdi Abaakouk wrote:

And my bench seems to confirm the perf issue have been solved:


I have updated my requirement review to require >=1.3.1 [1] to solve
the monasca issue.

[1] https://review.openstack.org/404878

--
Mehdi Abaakouk
mail: sil...@sileht.net
irc: sileht

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Mehdi Abaakouk]

On Fri, Dec 02, 2016 at 03:29:59PM +1100, Tony Breeds wrote:

On Thu, Dec 01, 2016 at 04:52:52PM +, Keen, Joe wrote:


Unfortunately there’s nothing wrong on the Monasca side so far as we know.
 We test new versions of the kafka-python library outside of Monasca
before we bother to try integrating a new version.  Since 1.0 the
kafka-python library has suffered from crashes and memory leaks severe
enough that we’ve never attempted using it in Monasca itself.  We reported
the bugs we found to the kafka-python project but they were closed once
they released a new version.


So Opening bugs isn't working.  What about writing code?


The bug https://github.com/dpkp/kafka-python/issues/55

Reopening it would be the right solution here.

I can't reproduce the segfault neither and I agree with dpkp, that looks like a
ujson issue.

And my bench seems to confirm the perf issue have been solved:
(but not in the pointed version...)

$ pifpaf run kafka python kafka_test.py
kafka-python version: 0.9.5
...
fetch size 179200 -> 45681.8728864 messages per second
fetch size 204800 -> 47724.3810674 messages per second
fetch size 230400 -> 47209.9841092 messages per second
fetch size 256000 -> 48340.7719787 messages per second
fetch size 281600 -> 49192.9896743 messages per second
fetch size 307200 -> 50915.3291133 messages per second

$ pifpaf run kafka python kafka_test.py
kafka-python version: 1.0.2

fetch size 179200 -> 8546.77931323 messages per second
fetch size 204800 -> 9213.30958314 messages per second
fetch size 230400 -> 10316.668006 messages per second
fetch size 256000 -> 11476.2285269 messages per second
fetch size 281600 -> 12353.7254386 messages per second
fetch size 307200 -> 13131.2367288 messages per second

(1.1.1 and 1.2.5 have also the same issue)

$ pifpaf run kafka python kafka_test.py
kafka-python version: 1.3.1
fetch size 179200 -> 44636.9371873 messages per second
fetch size 204800 -> 44324.7085365 messages per second
fetch size 230400 -> 45235.8283208 messages per second
fetch size 256000 -> 45793.1044121 messages per second
fetch size 281600 -> 44648.6357019 messages per second
fetch size 307200 -> 44877.8445987 messages per second
fetch size 332800 -> 47166.9176281 messages per second
fetch size 358400 -> 47391.0057622 messages per second

Looks like it works well now :)


Just in case I have updated a bit the bench script to ensure it always works 
for me:

--- kafka_test.py.ori   2016-12-02 09:16:10.570677010 +0100
+++ kafka_test.py   2016-12-02 09:06:04.870370438 +0100
@@ -14,7 +14,7 @@
import time
import ujson

-KAFKA_URL = '92.168.10.6:9092'
+KAFKA_URL = '127.0.0.1:9092'
KAFKA_GROUP = 'kafka_python_perf'
KAFKA_TOPIC = 'raw-events'

@@ -24,6 +24,7 @@

def write():
k_client = KafkaClient(KAFKA_URL)
+k_client.ensure_topic_exists(KAFKA_TOPIC)
p = KeyedProducer(k_client,
  async=False,
  req_acks=KeyedProducer.ACK_AFTER_LOCAL_WRITE,


--
Mehdi Abaakouk
mail: sil...@sileht.net
irc: sileht

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Mehdi Abaakouk]

On Fri, Dec 02, 2016 at 03:19:26PM +1100, Tony Breeds wrote:

On Thu, Dec 01, 2016 at 07:57:37AM +0100, Mehdi Abaakouk wrote:
I think the solution is pretty simple. Just Fix it.  I'm not saying it's easy
but it is *simple*.  We're a group of skilled individuals We have several ways
forward which just haven't been done in the last 8 months.  Waiting doesn't seem
to be a viable course anymore.


I agree the solution is *simple*.


From my POV this really feels more like a social problem then a technical one.
I'd like to remind everyone with a technical stake in this that it's OpenStack
First, Project Team Second[1].  The thing that's best for OpenStack is for the
oslo.mesaging and monasca teams to come together and find a path forward.  It's
really great that this thread has started that process!


No they are no social issue here, I always tried to fix other projects
issues even they are not in my area of expertise when I want to go
forward. But I'm realistic, I can only do this only if the invest and
the time it requires is not too big. And to fix (by code) that one, this
is just too high for me, sorry.

Cheers,
--
Mehdi Abaakouk
mail: sil...@sileht.net
irc: sileht

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Tony Breeds]

On Thu, Dec 01, 2016 at 04:52:52PM +, Keen, Joe wrote:

> Unfortunately there’s nothing wrong on the Monasca side so far as we know.
>  We test new versions of the kafka-python library outside of Monasca
> before we bother to try integrating a new version.  Since 1.0 the
> kafka-python library has suffered from crashes and memory leaks severe
> enough that we’ve never attempted using it in Monasca itself.  We reported
> the bugs we found to the kafka-python project but they were closed once
> they released a new version.

So Opening bugs isn't working.  What about writing code?

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Tony Breeds]

On Thu, Dec 01, 2016 at 08:41:54AM -0800, Joshua Harlow wrote:
> Keen, Joe wrote:
> > I¹ll look into testing the newest version of kafka-python and see if it
> > meets our needs.  If it still isn¹t stable and performant enough what are
> > the available options?
> 
> Fix the kafka-python library or fix monasca; those seem to be the options to
> me :)

Yup, Also worth including fix oslo.messaging to meet monasca's needs.  But
*something* needs fixing.
 
> I'd also not like to block the rest of the world (from using newer versions
> of kafka-python) during this as well. But then this may diverge/expand into
> a discussion we had a few summits ago, about getting rid of
> co-instability...

lalalalala not listening ;P

Less flippantly, there are a couple of ways to do this but IMO they're not in
the best interest of OpenStack.

1. vendor/fork python-kafka 0.X
2. Stop the proposal-bot from syncing with monasca, thereby allowing it to use
python-kafka 0.X at the expense of co-installability.

Fortunately either option is easy to reverse once the underlying issue is fixed.

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Tony Breeds]

On Thu, Dec 01, 2016 at 07:57:37AM +0100, Mehdi Abaakouk wrote:
> I'm aware of all of that, oslo.messaging patch for the new version is
> ready since 8 months now. And we are still blocked... capping libraries,
> whatever the reason, is very annoying and just freezes people work.

And I agree with that statement.  Capping isn't the right solution.  By the
same token uncapping isn't eaither as we know we'll cause breakage.

> From the API pov python-kafka haven't break anything, the API is still
> here and documented (and deprecated). What monasca raises is performance
> issue due to how their uses the library, and on absumption on how it works
> internally. Blocking all projects for that looks not fair to me.

I understand that your blocked, but blocking the other team isn't the solution
either.
 
> As nadya said, now, we have users that that prefers using an unmerged
> patch and the new lib instead of using upstream supported
> version with the old lib. This is not an acceptable situation to me but
> that's just my thought.
> 
> Where is the solution to allow oslo.messaging works blocked since 8
> month to continue ?
> 
> So, I'm waiting for monasca team input now and hope we can move forward.

I think the solution is pretty simple. Just Fix it.  I'm not saying it's easy
but it is *simple*.  We're a group of skilled individuals We have several ways
forward which just haven't been done in the last 8 months.  Waiting doesn't seem
to be a viable course anymore.

From my POV this really feels more like a social problem then a technical one.
I'd like to remind everyone with a technical stake in this that it's OpenStack
First, Project Team Second[1].  The thing that's best for OpenStack is for the
oslo.mesaging and monasca teams to come together and find a path forward.  It's
really great that this thread has started that process!

Other people are much better placed to define the way forward, clearly fixing
python-kafka is the best option which almost certainly means joining that
community.

Worst case if we really can't find a way forward, monasca can just vendor/fork
python-kafka 0.X and we'll all admit defeat.

Yours Tony.

[1] 
http://governance.openstack.org/reference/principles.html#openstack-first-project-team-second-company-third


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Keen, Joe]

On 12/1/16, 9:44 AM, "Julien Danjou"  wrote:

>On Thu, Dec 01 2016, Keen, Joe wrote:
>
>Hi Joe,
>
>[…]
>
>> The message context wrapped around every message is extra overhead we
>> don¹t want.  
>> There¹s no support for batch sends to Kafka.
>> There¹s no support for keyed producers.
>> The forced auto_commit on consumers isn¹t something we can tolerate.
>> There is no auto balance of multiple consumers in the same consumer
>>group
>> on a topic.
>
>This sounds like interesting features and optimizations to add to
>oslo.messaging. Did you already send patches or spec to oslo.messaging
>to improve that? What there the issues you encountered to fix those
>problems / add those new abilities?
>
>-- 
>Julien Danjou
>// Free Software hacker
>// https://julien.danjou.info


I haven’t submitted any patches to oslo.messaging.  When I wrote the Kafka
interfaces for Monasca I didn’t know oslo.messaging existed.  Since then I
just haven’t had the time to try merging the two approaches into one
library.  If anyone else is interested in what we’ve done I can help with
that.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Joshua Harlow]

Reopen them?

If they weren't fixed, then seems like the right thing to try to do, and
rinse and repeat until actually fixed :-P

Keen, Joe wrote:
> We reported
> the bugs we found to the kafka-python project but they were closed once
> they released a new version.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Joshua Harlow]

Damn, that's supposed to say co-installability...

Lol,

-Josh

Joshua Harlow wrote:

Keen, Joe wrote:

I¹ll look into testing the newest version of kafka-python and see if it
meets our needs. If it still isn¹t stable and performant enough what are
the available options?


Fix the kafka-python library or fix monasca; those seem to be the
options to me :)

I'd also not like to block the rest of the world (from using newer
versions of kafka-python) during this as well. But then this may
diverge/expand into a discussion we had a few summits ago, about getting
rid of co-instability...

-Josh

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Keen, Joe]

On 12/1/16, 9:41 AM, "Joshua Harlow"  wrote:

>Keen, Joe wrote:
>> I¹ll look into testing the newest version of kafka-python and see if it
>> meets our needs.  If it still isn¹t stable and performant enough what
>>are
>> the available options?
>
>Fix the kafka-python library or fix monasca; those seem to be the
>options to me :)
>
>I'd also not like to block the rest of the world (from using newer
>versions of kafka-python) during this as well. But then this may
>diverge/expand into a discussion we had a few summits ago, about getting
>rid of co-instability...
>
>-Josh

Unfortunately there’s nothing wrong on the Monasca side so far as we know.
 We test new versions of the kafka-python library outside of Monasca
before we bother to try integrating a new version.  Since 1.0 the
kafka-python library has suffered from crashes and memory leaks severe
enough that we’ve never attempted using it in Monasca itself.  We reported
the bugs we found to the kafka-python project but they were closed once
they released a new version.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Julien Danjou]

On Thu, Dec 01 2016, Keen, Joe wrote:

Hi Joe,

[…]

> The message context wrapped around every message is extra overhead we
> don¹t want.  
> There¹s no support for batch sends to Kafka.
> There¹s no support for keyed producers.
> The forced auto_commit on consumers isn¹t something we can tolerate.
> There is no auto balance of multiple consumers in the same consumer group
> on a topic.

This sounds like interesting features and optimizations to add to
oslo.messaging. Did you already send patches or spec to oslo.messaging
to improve that? What there the issues you encountered to fix those
problems / add those new abilities?

-- 
Julien Danjou
// Free Software hacker
// https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Joshua Harlow]

Keen, Joe wrote:

I¹ll look into testing the newest version of kafka-python and see if it
meets our needs.  If it still isn¹t stable and performant enough what are
the available options?


Fix the kafka-python library or fix monasca; those seem to be the 
options to me :)


I'd also not like to block the rest of the world (from using newer 
versions of kafka-python) during this as well. But then this may 
diverge/expand into a discussion we had a few summits ago, about getting 
rid of co-instability...


-Josh

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Keen, Joe]

On 12/1/16, 3:11 AM, "Julien Danjou"  wrote:

>On Thu, Dec 01 2016, Mehdi Abaakouk wrote:
>
>> I'm aware of all of that, oslo.messaging patch for the new version is
>> ready since 8 months now. And we are still blocked... capping libraries,
>> whatever the reason, is very annoying and just freezes people work.
>>
>> From the API pov python-kafka haven't break anything, the API is still
>> here and documented (and deprecated). What monasca raises is performance
>> issue due to how their uses the library, and on absumption on how it
>>works
>> internally. Blocking all projects for that looks not fair to me.
>>
>> As nadya said, now, we have users that that prefers using an unmerged
>> patch and the new lib instead of using upstream supported
>> version with the old lib. This is not an acceptable situation to me but
>>that's
>> just my thought.
>>
>> Where is the solution to allow oslo.messaging works blocked since 8
>> month to continue ?
>
>+1
>
>And if Monasca is using messaging, I wonder why they don't rely on
>oslo.messaging, which would also solve this entire problem in a snap.


Julien, I looked at the current oslo.messaging Kafka driver and
unfortunately I don¹t think it meets our needs.  There are several major
pieces that are a problem.

The message context wrapped around every message is extra overhead we
don¹t want.  
There¹s no support for batch sends to Kafka.
There¹s no support for keyed producers.
The forced auto_commit on consumers isn¹t something we can tolerate.
There is no auto balance of multiple consumers in the same consumer group
on a topic.

The current Kafka driver we use in monasca-common has all these features.
Without them we can¹t meet the durability and performance levels we need.

What sort of performance levels is oslo.messaging tested at?

I¹ll look into testing the newest version of kafka-python and see if it
meets our needs.  If it still isn¹t stable and performant enough what are
the available options?


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Julien Danjou]

On Thu, Dec 01 2016, Mehdi Abaakouk wrote:

> I'm aware of all of that, oslo.messaging patch for the new version is
> ready since 8 months now. And we are still blocked... capping libraries,
> whatever the reason, is very annoying and just freezes people work.
>
> From the API pov python-kafka haven't break anything, the API is still
> here and documented (and deprecated). What monasca raises is performance
> issue due to how their uses the library, and on absumption on how it works
> internally. Blocking all projects for that looks not fair to me.
>
> As nadya said, now, we have users that that prefers using an unmerged
> patch and the new lib instead of using upstream supported
> version with the old lib. This is not an acceptable situation to me but that's
> just my thought.
>
> Where is the solution to allow oslo.messaging works blocked since 8
> month to continue ?

+1

And if Monasca is using messaging, I wonder why they don't rely on
oslo.messaging, which would also solve this entire problem in a snap.

-- 
Julien Danjou
-- Free Software hacker
-- https://julien.danjou.info


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Mehdi Abaakouk]

Le 2016-12-01 01:25, Tony Breeds a écrit :
So right now I think the answer is still no we can't uncap it.  To the 
best of
my knowledge monasca still suffers a substatial performance decrease 
and the

switch from sync to async producers isn't desired for them.

Joe wrote this all up in:

http://lists.openstack.org/pipermail/openstack-dev/2016-June/098420.html


Unfortunately this comes down to monasca and olso.messaging getting on 
the same
page with python-kafka.  From a requirements POV we really only care 
about

co-installability, which uncappiugn clearly breaks.


I'm aware of all of that, oslo.messaging patch for the new version is
ready since 8 months now. And we are still blocked... capping libraries,
whatever the reason, is very annoying and just freezes people work.


From the API pov python-kafka haven't break anything, the API is still

here and documented (and deprecated). What monasca raises is performance
issue due to how their uses the library, and on absumption on how it works
internally. Blocking all projects for that looks not fair to me.

As nadya said, now, we have users that that prefers using an unmerged
patch and the new lib instead of using upstream supported
version with the old lib. 
This is not an acceptable situation to me but that's just my thought.


Where is the solution to allow oslo.messaging works blocked since 8
month to continue ?

So, I'm waiting for monasca team input now and hope we can move forward.

Cheers,
--
Mehdi Abaakouk
mail: sil...@sileht.net
irc: sileht

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [oslo][monasca] Can we uncap python-kafka ?

[Tony Breeds]

On Wed, Nov 30, 2016 at 08:13:44PM +0100, Mehdi Abaakouk wrote:
> 
> 
> Le 2016-11-30 19:11, Nadya Shakhat a écrit :
> > Hi all,
> > 
> > This one [1] is used very intensively actually :) It is not merged
> > because
> > we cannot just upgrade python-kafka version because of Monasca. We had
> > some
> > talks in their channel, but no results still. I suggest the following:
> > 1. On our side we will update this change
> > 2. I will ping Monasca guys once more and we will proceed with
> > discussions
> > in the CR
> > 
> > [1] https://review.openstack.org/#/c/332105/
> 
> Good news, so I should retitle this thread "Can we uncap python-kafka ?"

So right now I think the answer is still no we can't uncap it.  To the best of
my knowledge monasca still suffers a substatial performance decrease and the
switch from sync to async producers isn't desired for them.

Joe wrote this all up in:
http://lists.openstack.org/pipermail/openstack-dev/2016-June/098420.html

Unfortunately this comes down to monasca and olso.messaging getting on the same
page with python-kafka.  From a requirements POV we really only care about
co-installability, which uncappiugn clearly breaks.

Yours Tony.


signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev