We've added our production deployment experience for Kafka and Zookeeper here - https://cwiki.apache.org/confluence/display/KAFKA/Operations
Thanks, Neha On Tue, Oct 30, 2012 at 8:07 AM, Matthew Rathbone <matt...@foursquare.com> wrote: > When thinking about memory for the broker, the only thing you should > consider is the filesystem cache. The further behind production you're > consuming, the more memory matters (eg keeping your cache window larger > than the gap between production and consumption). > > As a caveat, this is only important if you're really thrashing the brokers > hard, we don't even see a blip if we consume from disk and we're pushing > them as hard as we can :-). > > > > On Tue, Oct 30, 2012 at 8:21 AM, Jun Rao <jun...@gmail.com> wrote: > >> Yes. >> >> Thanks, >> >> Jun >> >> On Mon, Oct 29, 2012 at 9:34 PM, howard chen <howac...@gmail.com> wrote: >> >> > We only a few internal consumers, so assume they should be fine? >> > >> > >> > On Tue, Oct 30, 2012 at 12:27 PM, Jun Rao <jun...@gmail.com> wrote: >> > > Kafka broker typically doesn't need a lot memory. So 2GB is fine. ZK >> > memory >> > > depends on # consumers. More consumers mean more offsets written to ZK. >> > > >> > > Thanks, >> > > >> > > Jun >> > > >> > > On Mon, Oct 29, 2012 at 9:07 PM, howard chen <howac...@gmail.com> >> wrote: >> > > >> > >> I understand the main limitation of Kafka deployment is the disk >> space, >> > >> >> > >> E.g. >> > >> >> > >> If I generate 10GB message per day, and I have 2 nodes, and I need to >> > >> keep for 10 days, then I need >> > >> >> > >> 10GB * 10 / 2 = 50GB per node (of course there are overhead, but the >> > >> requirement is somehow proportional.) >> > >> >> > >> So If I deploy machinesusing the following setup, do you think they >> > >> are reasonable? >> > >> >> > >> 2 x Kafka (100GB, 2xCPU x 2GB RAM) >> > >> 3 x Zookeeper ( 10GB, 1x CPU, 512MB RAM) >> > >> >> > >> do you think it is okay? >> > >> >> > >> > > > > -- > Matthew Rathbone > Foursquare | Software Engineer | Server Engineering Team > matt...@foursquare.com | @rathboma <http://twitter.com/rathboma> | > 4sq<http://foursquare.com/rathboma>
