will surely do that. Initially this will complement existing scribe aggregators, and later might replace the aggregators.
thanks, Raghu. On Wed, Apr 25, 2012 at 7:08 PM, Jun Rao <jun...@gmail.com> wrote: > Raghu, > > It's good to hear that Twitter will start to use Kafka in prod soon. Once > that happens, do you think you can add your usage in the powerby wiki page? > > Thanks, > > Jun > > On Wed, Apr 25, 2012 at 6:31 PM, Raghu Angadi <raghuang...@gmail.com> > wrote: > > > another factor to be aware of is the effect of partitions. Even with > async > > producer in my use case, I saw lower compression ratio because the > messages > > are randomly split across 5 partitions.. A simple work around was to use > a > > key such that each of 1000 consecutive messages go to one partition. > > > > we are really happy with how well Kafka 'just worked' in our prototype at > > twitter. will likely use it production soon.. > > > > On Wed, Apr 25, 2012 at 10:47 AM, Felix GV <fe...@mate1inc.com> wrote: > > > > > Also, compression ratios are usually better on larger payloads, so > > > compression should, in most cases, be more effective when combined with > > > async because the batching of messages results in larger payloads, and > > thus > > > better compression ratios. > > > > > > -- > > > Felix > > > > > > > > > > > > On Wed, Apr 25, 2012 at 8:14 AM, Joel Koshy <jjkosh...@gmail.com> > wrote: > > > > > > > Hi Navneet, > > > > > > > > From docs, its not clear whether async mode and compression are > related > > > or > > > > > independent. Can you please elaborate? > > > > > > > > > > > > > They are related in the sense that the DefaultEventHandler that is > used > > > in > > > > async mode compresses a batch of messages when they are serialized. > > > > However, compression is not tied to async mode. Sync mode can also > use > > > > compression, but it does not do batching. > > > > > > > > > > > > > > > > > > One more thing - compression definitely enhances performance, but > is > > it > > > > > true for async mode as well (assuming they are not related)? > > > > > > > > > > > > > Async mode basically enables batching of messages so its throughput > > gain > > > is > > > > orthogonal to that gained by compression. > > > > > > > > Thanks, > > > > > > > > Joel > > > > > > > > > >
