Thanks Neha and Jun for the pointers. We will try and evaluate this as well.
Hemanth On Sat, Sep 13, 2014 at 4:45 AM, Neha Narkhede <neha.narkh...@gmail.com> wrote: > Hemanth, > > Specifically, you'd want to monitor > kafka:type=kafka.SocketServerStats:getMaxProduceRequestMs and > kafka:type=kafka.LogFlushStats:getMaxFlushMs. If the broker is under load > due to frequent flushes, it will almost certainly show up as spikes in the > flush latency and consequently the produce request latency. A side effect > of that is that your producer queue will back up and your producer will > eventually lose data. > > Thanks, > Neha > > On Thu, Sep 11, 2014 at 5:48 PM, Hemanth Yamijala <yhema...@gmail.com> > wrote: > > > Neha, > > > > Thanks. We are on 0.7.2. I have written on another thread on the list > here > > about one of the reasons we are stuck - the absence of a PHP client for > our > > front end producer systems. (On a side note, would appreciate if any > inputs > > can be given on that thread as well) > > > > When you mean performance, do you mean throughput ? We did measure > > throughput with our default configuration of 1000 ms for the flush > interval > > value, and the much lower 100 ms value I proposed on this thread. Our > > numbers were identical - for a single broker we were clocking at around > > 20,000 messages read per second on the consumer side. Using a small 'n' > > brokers we can easily exceed our target numbers. (The load was > > synthetically generated - using a likely message size and at a rate that > > seems reasonable for our producing side). > > > > Given this observation, do you suggest any further tests / measurements > for > > us to be sure ? Would appreciate any inputs. > > > > Thanks > > Hemanth > > > > On Fri, Sep 12, 2014 at 1:32 AM, Neha Narkhede <neha.narkh...@gmail.com> > > wrote: > > > > > I should mention that the impact of doing so is much higher wrt to > > taking a > > > hit on performance, on versions < 0.8.1. As long as you're on 0.8.1 or > > > later, it should mostly be fine. You might want to keep a close tab on > > how > > > your iostat numbers are doing, to be sure. > > > > > > On Wed, Sep 10, 2014 at 5:46 PM, Hemanth Yamijala <yhema...@gmail.com> > > > wrote: > > > > > > > Thanks Jun. > > > > > > > > On Thu, Sep 11, 2014 at 4:13 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > > > As long as the I/O load is reasonable, this is probably ok. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Wed, Sep 10, 2014 at 4:59 AM, Hemanth Yamijala < > > yhema...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi folks, > > > > > > > > > > > > In order to meet latency requirements for a system we are > building, > > > we > > > > > > tested with different values of the above two parameters and > found > > > that > > > > > > settings as low as 100 work best for us, balancing the required > > > > > throughput > > > > > > and latencies. > > > > > > > > > > > > I just wanted to check if 100 is a sane value, notwithstanding we > > are > > > > > > getting good results in our tests, anything we need to be aware > of > > > > while > > > > > > setting to low values like this (apart from the throughput, which > > we > > > > see > > > > > is > > > > > > OK for us) ? > > > > > > > > > > > > Any experience reports will help. > > > > > > > > > > > > Thanks > > > > > > Hemanth > > > > > > > > > > > > > > > > > > > > >
