Raghu, It's good to hear that Twitter will start to use Kafka in prod soon. Once that happens, do you think you can add your usage in the powerby wiki page?
Thanks, Jun On Wed, Apr 25, 2012 at 6:31 PM, Raghu Angadi <raghuang...@gmail.com> wrote: > another factor to be aware of is the effect of partitions. Even with async > producer in my use case, I saw lower compression ratio because the messages > are randomly split across 5 partitions.. A simple work around was to use a > key such that each of 1000 consecutive messages go to one partition. > > we are really happy with how well Kafka 'just worked' in our prototype at > twitter. will likely use it production soon.. > > On Wed, Apr 25, 2012 at 10:47 AM, Felix GV <fe...@mate1inc.com> wrote: > > > Also, compression ratios are usually better on larger payloads, so > > compression should, in most cases, be more effective when combined with > > async because the batching of messages results in larger payloads, and > thus > > better compression ratios. > > > > -- > > Felix > > > > > > > > On Wed, Apr 25, 2012 at 8:14 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > > > Hi Navneet, > > > > > > From docs, its not clear whether async mode and compression are related > > or > > > > independent. Can you please elaborate? > > > > > > > > > > They are related in the sense that the DefaultEventHandler that is used > > in > > > async mode compresses a batch of messages when they are serialized. > > > However, compression is not tied to async mode. Sync mode can also use > > > compression, but it does not do batching. > > > > > > > > > > > > > > One more thing - compression definitely enhances performance, but is > it > > > > true for async mode as well (assuming they are not related)? > > > > > > > > > > Async mode basically enables batching of messages so its throughput > gain > > is > > > orthogonal to that gained by compression. > > > > > > Thanks, > > > > > > Joel > > > > > >
