Oops, missed what you said--that you had already dropped the flush interval. Listen to Neha :-)
-Jay On Thu, Nov 8, 2012 at 7:57 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > Hi Bob, > > Currently the broker does not hand out messages to consumers until they > are flushed to disk, this means the flush interval acts as a lower bound on > worst case latency. Setting that lower should fix the problem. > > This problem has been eliminated in the next release, as both the blocking > on flush and the fetcher backoff have been eliminated--this should drop > latency to a few ms. > > -Jay > > > On Wed, Nov 7, 2012 at 5:55 PM, Bob Cotton <bcot...@rallydev.com> wrote: > >> Hello, >> >> We have a low-volume topic (~75msgs/sec) for which we would like to have a >> low propagation delay from producer to consumer. >> >> We have 3 brokers, each with a default of 4 partitions each. for a total >> of >> 12 partitions. >> The producer is sync, without compression. There are 8 producers each >> producing 1/8 of the traffic. >> We are using the high-level java consumer, with 4 threads consuming the >> topic. >> >> We are wrapping the message with a custom Encoder/Decoder and record >> currentTimeMillis() on the sender, and do the same in the receiver, then >> record the propagation delay. All hosts are time synced with ntp. >> >> With the settings on the broker for flush messages and flush interval >> (unset, defaults to 500 msgs and 3000ms) the overall 95th percentile for >> propagation is 2,500ms. >> >> When we adjust the topic flush interval to 20ms, the 95th percentile drops >> to 1,700ms >> When we adjust the consumers "fetcher.backoff.ms" to 10, the 95th >> percentile drops to about 970ms. >> >> We would like this to be sub-500ms. >> We could run with less partitions and/or more consumer threads. >> >> Anything glaring about this config? anything we're missing? >> >> Thanks >> -Bob >> > >
