: SenthilKumar K [mailto:senthilec...@gmail.com]
>>> Sent: Wednesday, June 21, 2017 10:28 AM
>>> To: Tauzell, Dave
>>> Cc: users@kafka.apache.org; senthilec...@apache.org;
>>> d...@kafka.apache.org
>>> Subject: Re: Handling 2 to 3 Million Events befor
umar K [mailto:senthilec...@gmail.com senthilec...@gmail.com>]
> Sent: Wednesday, June 21, 2017 8:55 AM
> To: Tauzell, Dave
> Cc: users@kafka.apache.org<mailto:users@kafka.apache.org>;
> senthilec...@apache.org<mailto:senthilec...@apache.org>;
> d...@kafka.ap
...@kafka.apache.org
Subject: Re: Handling 2 to 3 Million Events before Kafka
So netty would work for this case ? I do have netty server and seems to be i'm
not getting the expected results .. here is the git
https://github.com/senthilec566/netty4-server , is this right implementation ?
Cheers
> *To:* Tauzell, Dave
> *Cc:* users@kafka.apache.org; senthilec...@apache.org;
> d...@kafka.apache.org; Senthil kumar
> *Subject:* Re: Handling 2 to 3 Million Events before Kafka
>
>
>
> Thanks Jeyhun. Yes http server would be problematic here w.r.t network ,
> memory ..
>
.
It seems possible with the right sort of kafka producer tuning.
-Dave
From: SenthilKumar K [mailto:senthilec...@gmail.com]
Sent: Wednesday, June 21, 2017 8:55 AM
To: Tauzell, Dave
Cc: users@kafka.apache.org; senthilec...@apache.org; d...@kafka.apache.org;
Senthil kumar
Subject: Re: Handling 2 to 3
Thanks Jeyhun. Yes http server would be problematic here w.r.t network ,
memory ..
Hi Dave , The problem is not with Kafka , it's all about how do you handle
huge data before kafka. I did a simple test with 5 node Kafka Cluster
which gives good result ( ~950 MB/s ) ..So Kafka side i dont see a
What are your configurations?
- production
- brokers
- consumers
Is the problem that web servers cannot send to Kafka fast enough or your
consumers cannot process messages off of kafka fast enough?
What is the average size of these messages?
-Dave
-Original Message-
From: SenthilKumar
Hi,
With kafka you can increase overall throughput by increasing the number of
nodes in a cluster.
I had a similar issue, where we needed to ingest vast amounts of data to
streaming system.
In our case, kafka was a bottleneck, because of disk I/O. To solve it, we
implemented (simple) distributed
