Re: Handling 2 to 3 Million Events before Kafka

2017-06-22 Thread SenthilKumar K
: SenthilKumar K [mailto:senthilec...@gmail.com] >>> Sent: Wednesday, June 21, 2017 10:28 AM >>> To: Tauzell, Dave >>> Cc: users@kafka.apache.org; senthilec...@apache.org; >>> d...@kafka.apache.org >>> Subject: Re: Handling 2 to 3 Million Events befor

Re: Handling 2 to 3 Million Events before Kafka

2017-06-21 Thread Garrett Barton
umar K [mailto:senthilec...@gmail.com senthilec...@gmail.com>] > Sent: Wednesday, June 21, 2017 8:55 AM > To: Tauzell, Dave > Cc: users@kafka.apache.org<mailto:users@kafka.apache.org>; > senthilec...@apache.org<mailto:senthilec...@apache.org>; > d...@kafka.ap

RE: Handling 2 to 3 Million Events before Kafka

2017-06-21 Thread Tauzell, Dave
...@kafka.apache.org Subject: Re: Handling 2 to 3 Million Events before Kafka So netty would work for this case ? I do have netty server and seems to be i'm not getting the expected results .. here is the git https://github.com/senthilec566/netty4-server , is this right implementation ? Cheers

Re: Handling 2 to 3 Million Events before Kafka

2017-06-21 Thread SenthilKumar K
> *To:* Tauzell, Dave > *Cc:* users@kafka.apache.org; senthilec...@apache.org; > d...@kafka.apache.org; Senthil kumar > *Subject:* Re: Handling 2 to 3 Million Events before Kafka > > > > Thanks Jeyhun. Yes http server would be problematic here w.r.t network , > memory .. >

RE: Handling 2 to 3 Million Events before Kafka

2017-06-21 Thread Tauzell, Dave
a.apache.org> Subject: Handling 2 to 3 Million Events before Kafka Hi Team , Sorry if this question is irrelevant to Kafka Group ... I have been trying to solve problem of handling 5 GB/sec ingestion. Kafka is really good candidate for us to handle this ingestion rate .. 100K machi

Re: Handling 2 to 3 Million Events before Kafka

2017-06-21 Thread SenthilKumar K
mailto:senthilec...@gmail.com] > Sent: Wednesday, June 21, 2017 7:58 AM > To: users@kafka.apache.org > Cc: senthilec...@apache.org; Senthil kumar; d...@kafka.apache.org > Subject: Handling 2 to 3 Million Events before Kafka > > Hi Team , Sorry if this question is irrelevant to Kaf

RE: Handling 2 to 3 Million Events before Kafka

2017-06-21 Thread Tauzell, Dave
K [mailto:senthilec...@gmail.com] Sent: Wednesday, June 21, 2017 7:58 AM To: users@kafka.apache.org Cc: senthilec...@apache.org; Senthil kumar; d...@kafka.apache.org Subject: Handling 2 to 3 Million Events before Kafka Hi Team , Sorry if this question is irrelevant to Kafka Group ... I have

Re: Handling 2 to 3 Million Events before Kafka

2017-06-21 Thread Jeyhun Karimov
Hi, With kafka you can increase overall throughput by increasing the number of nodes in a cluster. I had a similar issue, where we needed to ingest vast amounts of data to streaming system. In our case, kafka was a bottleneck, because of disk I/O. To solve it, we implemented (simple) distributed

Handling 2 to 3 Million Events before Kafka

2017-06-21 Thread SenthilKumar K
Hi Team , Sorry if this question is irrelevant to Kafka Group ... I have been trying to solve problem of handling 5 GB/sec ingestion. Kafka is really good candidate for us to handle this ingestion rate .. 100K machines > { Http Server (Jetty/Netty) } --> Kafka Cluster.. I see the problem