So netty would work for this case ? I do have netty server and seems to be i'm not getting the expected results .. here is the git https://github.com/senthilec566/netty4-server , is this right implementation ?
Cheers, Senthil On Wed, Jun 21, 2017 at 7:45 PM, Tauzell, Dave <dave.tauz...@surescripts.com > wrote: > I see. > > 1. You don’t want the 100k machines sending directly to kafka. > > 2. You can only have a small number of web servers > > > > People certainly have web-servers handling over 100k concurrent > connections. See this for some examples: https://github.com/smallnest/ > C1000K-Servers . > > > > It seems possible with the right sort of kafka producer tuning. > > > > -Dave > > > > *From:* SenthilKumar K [mailto:senthilec...@gmail.com] > *Sent:* Wednesday, June 21, 2017 8:55 AM > *To:* Tauzell, Dave > *Cc:* users@kafka.apache.org; senthilec...@apache.org; > d...@kafka.apache.org; Senthil kumar > *Subject:* Re: Handling 2 to 3 Million Events before Kafka > > > > Thanks Jeyhun. Yes http server would be problematic here w.r.t network , > memory .. > > > > Hi Dave , The problem is not with Kafka , it's all about how do you > handle huge data before kafka. I did a simple test with 5 node Kafka > Cluster which gives good result ( ~950 MB/s ) ..So Kafka side i dont see a > scaling issue ... > > > > All we are trying is before kafka how do we handle messages from different > servers ... Webservers can send fast to kafka but still i can handle only > 50k events per second which is less for my use case.. also i can't deploy > 20 webservers to handle this load. I'm looking for an option what could be > the best candidate before kafka , it should be super fast in getting all > and send it to kafka producer .. > > > > > > --Senthil > > > > On Wed, Jun 21, 2017 at 6:53 PM, Tauzell, Dave < > dave.tauz...@surescripts.com> wrote: > > What are your configurations? > > - production > - brokers > - consumers > > Is the problem that web servers cannot send to Kafka fast enough or your > consumers cannot process messages off of kafka fast enough? > What is the average size of these messages? > > -Dave > > > -----Original Message----- > From: SenthilKumar K [mailto:senthilec...@gmail.com] > Sent: Wednesday, June 21, 2017 7:58 AM > To: users@kafka.apache.org > Cc: senthilec...@apache.org; Senthil kumar; d...@kafka.apache.org > Subject: Handling 2 to 3 Million Events before Kafka > > Hi Team , Sorry if this question is irrelevant to Kafka Group ... > > I have been trying to solve problem of handling 5 GB/sec ingestion. Kafka > is really good candidate for us to handle this ingestion rate .. > > > 100K machines ----> { Http Server (Jetty/Netty) } --> Kafka Cluster.. > > I see the problem in Http Server where it can't handle beyond 50K events > per instance .. I'm thinking some other solution would be right choice > before Kafka .. > > Anyone worked on similar use case and similar load ? Suggestions/Thoughts ? > > --Senthil > > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of the > individual or entity to whom they are addressed. If you have received this > e-mail in error, please notify the sender by reply e-mail immediately and > destroy all copies of the e-mail and any attachments. > > >
