Re: Apache Kafka in AWS

2013-05-29 Thread S Ahmed
Subject: Re: Apache Kafka in AWS Curious if you tested with larger message sizes, like around 20-30kb (you mentioned 2kb). Any numbers on that size? On Thu, May 23, 2013 at 10:12 AM, Jason Weiss jason_we...@rapid7.com wrote: Bummer. Yes, but it will be several days. I'll post back

RE: Apache Kafka in AWS

2013-05-29 Thread Jason Weiss
Nope, it was specific to the application I'm working on carrying real-world data. Sorry :-( Jason From: S Ahmed [sahmed1...@gmail.com] Sent: Wednesday, May 29, 2013 13:40 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Is the code you used

RE: Apache Kafka in AWS

2013-05-23 Thread Jason Weiss
constant. The flatline is pretty obvious in the extended 4 minute test-- it ramps up, flat lines, then ramps down. Jason From: Jun Rao [jun...@gmail.com] Sent: Thursday, May 23, 2013 00:17 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Jason

Re: Apache Kafka in AWS

2013-05-23 Thread Jason Weiss
[jun...@gmail.com] Sent: Thursday, May 23, 2013 00:17 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Jason, Thanks for sharing. This is very interesting. Normally, Kafka brokers don't use too much CPU. Are most of the 750% CPU actually used by Kafka brokers? Jun On Wed, May

Apache Kafka in AWS

2013-05-22 Thread Jason Weiss
All, I asked a number of questions of the group over the last week, and I'm happy to report that I've had great success getting Kafka up and running in AWS. I am using 3 EC2 instances, each of which is a M2 High-Memory Quadruple Extra Large with 8 cores and 58.4 GiB of memory according to the

Re: Apache Kafka in AWS

2013-05-22 Thread Neha Narkhede
Thanks for sharing your experience with the community, Jason! -Neha On Wed, May 22, 2013 at 1:42 PM, Jason Weiss jason_we...@rapid7.com wrote: All, I asked a number of questions of the group over the last week, and I'm happy to report that I've had great success getting Kafka up and

Re: Apache Kafka in AWS

2013-05-22 Thread Ken Krugler
Hi Jason, Thanks for the notes. I'm curious whether you went with using local drives (ephemeral storage) or EBS, and if with EBS then what IOPS. Thanks, -- Ken On May 22, 2013, at 1:42pm, Jason Weiss wrote: All, I asked a number of questions of the group over the last week, and I'm

RE: Apache Kafka in AWS

2013-05-22 Thread Jason Weiss
Ken, Great question! I should have indicated I was using EBS, 500GB with 2000 provisioned IOPs. Jason From: Ken Krugler [kkrugler_li...@transpac.com] Sent: Wednesday, May 22, 2013 17:23 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Hi

Re: Apache Kafka in AWS

2013-05-22 Thread Jonathan Hodges
, May 22, 2013 17:23 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Hi Jason, Thanks for the notes. I'm curious whether you went with using local drives (ephemeral storage) or EBS, and if with EBS then what IOPS. Thanks, -- Ken On May 22, 2013, at 1:42pm, Jason Weiss wrote

RE: Apache Kafka in AWS

2013-05-22 Thread Jason Weiss
@kafka.apache.org Subject: Re: Apache Kafka in AWS Awesome right up Jason! Very helpful as we are also looking to build a Kafka environment in AWS. I am curious, are you using Kafka 0.7.2 or 0.8 in your tests? Did you have just one EBS volume per broker instance or RAID 10 across EBS volumes per broker? Thanks

Re: Apache Kafka in AWS

2013-05-22 Thread Scott Clasen
Hey Jason, question what openjdk version did you have issues with? Im running kafka on it now and has been ok. Was it a crash only at load? Thanks SC On Wed, May 22, 2013 at 1:42 PM, Jason Weiss jason_we...@rapid7.com wrote: All, I asked a number of questions of the group over the last

RE: Apache Kafka in AWS

2013-05-22 Thread Jason Weiss
...@heroku.com] Sent: Wednesday, May 22, 2013 19:27 To: users Subject: Re: Apache Kafka in AWS Hey Jason, question what openjdk version did you have issues with? Im running kafka on it now and has been ok. Was it a crash only at load? Thanks SC On Wed, May 22, 2013 at 1:42 PM, Jason Weiss jason_we

Re: Apache Kafka in AWS

2013-05-22 Thread Scott Clasen
: Scott Clasen [sc...@heroku.com] Sent: Wednesday, May 22, 2013 19:27 To: users Subject: Re: Apache Kafka in AWS Hey Jason, question what openjdk version did you have issues with? Im running kafka on it now and has been ok. Was it a crash only at load? Thanks SC On Wed, May 22, 2013

RE: Apache Kafka in AWS

2013-05-22 Thread Jason Weiss
Did you check that you were using all cores? top was reporting over 750% Jason From: Ken Krugler [kkrugler_li...@transpac.com] Sent: Wednesday, May 22, 2013 20:59 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Hi Jason, On May 22, 2013

Re: Apache Kafka in AWS

2013-05-22 Thread Jun Rao
was reporting over 750% Jason From: Ken Krugler [kkrugler_li...@transpac.com] Sent: Wednesday, May 22, 2013 20:59 To: users@kafka.apache.org Subject: Re: Apache Kafka in AWS Hi Jason, On May 22, 2013, at 3:35pm, Jason Weiss wrote: Ken, Great