Hi All,
I am trying to do a simple benchmark test for Kafka using single broker,
producer and consumer however my consumer doesn't seem to receive all the
messages produced by the producer so not sure what is going on any help?
Here is the full description of the problem.
http://stackoverflow.com/
Hi,
I have the following setup.
Single Kafka broker and Zookeeper on Machine 1single Kafka producer on Machine 2
Single Kafka Consumer on Machine 3
When a producer client sends a message to the Kafka broker by pointing at the
Zookeeper Server the consumer doesn't seem to get the message right away
Sep 14, 2016 11:58 PM, Ali Akhtar ali.rac...@gmail.com
wrote:
It sounds like a network issue. Where are the 3 servers located / hosted?
On Thu, Sep 15, 2016 at 11:51 AM, kant kodali wrote:
Hi,
I have the following setup.
Single Kafka broker and Zookeeper on Machine 1single Kafka produce
and
kafka-producer.sh instead of the Node code?
On Thu, Sep 15, 2016 at 12:01 PM, kant kodali wrote:
> They are hosted on AWS and I dont think there are any network issues
> because I
> tried testing other Queuing systems with no issues however I am using a
> node.js
> client w
My goal is to test the throughput (#messages per second) given my setup and
with a data size of 1KB. if you guys already have some idea on these numbers
that would be helpful as well.
On Thu, Sep 15, 2016 12:24 AM, kant kodali kanth...@gmail.com
wrote:
172.* is all private ip's f
m4.xlarge
On Thu, Sep 15, 2016 12:33 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
What's the instance size that you're using? With 300k messages your single
broker might not be able to handle it.
On Thu, Sep 15, 2016 at 12:30 PM, kant kodali wrote:
My goal is to test the
wrote:
Lower the workload gradually, start from 10 messages, increase to 100, then
1000, and so on. See if it slows down as the workload increases. If so, you
need more brokers + partitions to handle the workload.
On Thu, Sep 15, 2016 at 12:42 PM, kant kodali wrote:
> m4.xlarge
>
&
erformance.
Here's a case study that should help:
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
Good luck, let us know how it goes
On Thu, Sep 15, 2016 at 1:42 PM, kant kodali wrote:
yeah..
I tried it with 1
with Kafka I tried it with 10 messages with single broker and only one partiton
that looked instantaneous and ~5K messages/sec for the data size of 1KB
I tried it with 1000 messages that looked instantaneous as well ~5K messages/sec
for the data size of 1KBI tried it with 10K messages with single b
:16 AM, Ben Davison ben.davi...@7digital.com
wrote:
Hi Kant,
I was following the other thread, can you try using a different
benchmarking client for a test.
https://grey-boundary.io/load-testing-apache-kafka-on-aws/
Ben
On Thursday, 15 September 2016, kant kodali wrote:
with
give that a go..but can you
please confirm that one partition on one broker should be able to handle 300K
messages of 1KB data size for each message?
Thanks,kant
On Thu, Sep 15, 2016 2:28 AM, kant kodali kanth...@gmail.com
wrote:
Hi Ben,
I can give that a try but can you tell me the suspici
which port should I use 9091 or 9092 or 2181 to send messages through kafka
when using a client Library?
I start kafka as follows:
sudo bin/zookeeper-server-start.sh config/zookeeper.propertiessudo
./bin/kafka-server-start.sh config/server.properties
and I dont see any process running on 9091 or
thing?
On Thu, Sep 15, 2016 4:24 AM, Ali Akhtar ali.rac...@gmail.com
wrote:
Examine server.properties and see which port you're using in there
On Thu, Sep 15, 2016 at 3:52 PM, kant kodali wrote:
which port should I use 9091 or 9092 or 2181 to send messages through kafka
when usin
ts on 2181.
On Thu, 15 Sep 2016 at 17:03 kant kodali wrote:
> I haven't changed anything from
> https://github.com/apache/kafka/blob/trunk/config/server.properties
> and it looks like it is pointing to zookeeper.
> Question:
> Does producer client need to point 9092 a
https://issues.apache.org/jira/browse/KAFKA-1793
It would be great to use Consul instead of Zookeeper for Kafka and I think it
would benefit Kafka a lot from the exponentially growing consul community.
can one topic be registered in multiple brokers? if so, which component of
kafka decides which broker should get the message for that particular topic?
Thanks!
ker will be the leader of 1 of the 3 partitions.
P.S how did the benchmarking go?
On Sat, Sep 17, 2016 at 1:36 PM, kant kodali wrote:
> can one topic be registered in multiple brokers? if so, which component of
> kafka decides which broker should get the message for that particular
> topic?
> Thanks!
why did Kafka choose pull instead of push for a consumer? push sounds like it
is more realtime to me than poll and also wouldn't poll just keeps polling even
when they are no messages in the broker causing more traffic? please enlighten
me
#x27;m only guessing here regarding if this is the reason:
Pull is much more sensible when a lot of data is pushed through. It allows
consumers consuming at their own pace, slow consumers do not slow the complete
system down.
--
Best regards,
Rad
On Sat, Sep 17, 2016 at 11:18 AM +0
your
thoughts.
On Sat, Sep 17, 2016 12:39 PM, Radoslaw Gruchalski ra...@gruchalski.com
wrote:
Kafka is not a queue. It’s a distributed commit log.
–
Best regards,
Radek Gruchalski
ra...@gruchalski.com
On September 17, 2016 at 9:23:09 PM, kant kodali (kanth...@gmail.com) wrote
Why does comcast needs to do better than 1-2 seconds?
On Sun, Sep 18, 2016 8:08 PM, Ghosh, Achintya (Contractor)
achintya_gh...@comcast.com
wrote:
Hi there,
We have an usecase where we do a lot of business logic to process each message
and sometime it takes 1-2 sec, so will be Kafka fi
Kafka
independent of Zookeeper alltogether. Maybe not worth spending time on
migrating to Consul in that case.
Cheers,
Jens
On Sat, Sep 17, 2016 at 10:38 PM Jennifer Fountain
wrote:
> +2 watching.
>
> On Sat, Sep 17, 2016 at 2:45 AM, kant kodali wrote:
>
> > https:/
est regards,
> > Radek Gruchalski
> > ra...@gruchalski.com
> >
> >
> > On September 17, 2016 at 9:49:43 PM, kant kodali (kanth...@gmail.com)
> wrote:
> >
> > Still it should be possible to implement using reactive streams right.
> > Could you
Does Kafka Sync/persist every message from a publisher by default? If not, What
settings should I change so I Sync every message?
producer/consumer to limit
latency, and that may also be enough to get a low enough latency. It would
certainly be interesting to compare the two, with the same hardware, and
with high load.
On Thu, Sep 22, 2016 at 6:01 PM kant kodali wrote:
> @Gerard Thanks for this. It looks good
@Fei Just curious why you guys are interested in using Kafka. I thought
alcatel-lucent usually create their own software no?
On Fri, Sep 23, 2016 10:36 PM, Kamal C kamaltar...@gmail.com
wrote:
Reduce the metadata refresh interval 'metadata.max.age.ms' from 5 min to
your desired time interva
is there a way to make sure two consumers receive the same message from the
kafka broker in a atomic way? such that if consumer 1 gets a message
consumer 2 should also get that message and in case one of the consumer
fails for whatever reason consumer 2 should also rollback to previous
offset or in
Sorry there is a typo. here is a restatement.
Is there a way to make sure two consumers receive the same message from the
kafka broker in a atomic way? such that if consumer 1 gets a message
consumer 2 should also get that message and if consumer 1 fails for
whatever reason consumer 2 should also
u can
> write consumer 1 and consumer 2 to share a common external offset storage
> then you may be able to build the functionality you seek.
>
> -hans
>
>
>
> > On Nov 5, 2016, at 3:55 PM, kant kodali wrote:
> >
> > Sorry there is a typo. here is a restatemen
2016 at 4:15 PM, kant kodali wrote:
> Hi Hans,
>
> What do you mean by "write consumer 1 and consumer 2 to share a common
> external offset storage" ? can you please elaborate a bit more.
>
> Thanks!
>
> On Sat, Nov 5, 2016 at 4:00 PM, Hans Jespersen wrote:
>
tter way to get you the functionality you want?
>
> -hans
>
>
>
>
> > On Nov 5, 2016, at 4:31 PM, kant kodali wrote:
> >
> > I am new to Kafka and reading this statement "write consumer 1 and
> consumer
> > 2 to share a common external offset storage"
Hi! Thanks. any pointers on how to do that?
On Sun, Nov 6, 2016 at 2:32 PM, Tauzell, Dave
wrote:
> You should have one consumer pull the message and submit the data to each
> storage using an XA transaction.
>
> > On Nov 5, 2016, at 19:49, kant kodali wrote:
> >
>
Hi Hans,
The two storages we use are Cassandra and Elastic search and they are on
the same datacenter for now.
The Programming Language we use is Java and OS would be Ubuntu or CentOS.
We get messages in JSON format so we insert into Elastic Search directly
and for Cassandra we transform JSON mess
h...@confluent.io (650)924-2670
> */
>
> On Sun, Nov 6, 2016 at 7:08 PM, kant kodali wrote:
>
> > Hi Hans,
> >
> > The two storages we use are Cassandra and Elastic search and they are on
> > the same datacenter for now.
> > The Programming Language
rsion field as cassandra's
> writetime?(_version is strictly increasing, cassandra uses writetime for
> applying LWW, so last write in elasticsearch will always win)
>
> It needs no transaction and makes databases convergent.
>
>
> ____
> From: kant
fka ... I don't think this is possible.
>
> -Dave
>
> -Original Message-
> From: kant kodali [mailto:kanth...@gmail.com]
> Sent: Monday, November 7, 2016 10:48 AM
> To: users@kafka.apache.org
> Subject: Re: is there a way to make sure two consumers receive the
Fixing typo's
Hi Tauzell,
Yeah our users want to query, do aggregations on Elastic Search directly
and we cannot have inconsistent data because say the writes didn't make it
into Cassandra but made it to Elastic search then a simple aggregations
like count will lead to a wrong answer but again a
And there is this https://github.com/vroyer/elassandra which is still under
active development and not sure how they plan to keep up with Apache
Cassandra moving forward.
On Mon, Nov 7, 2016 at 9:36 AM, kant kodali wrote:
> Fixing typo's
>
> Hi Tauzell,
>
> Yeah our use
er writetime will win.
>
> Using es's document _version field is just one option, you can use
> something from you domain or kafka's offset or machine timestamp (not
> recommended at all).
>
> I hope it could help
>
> From: kant ko
Hi All,
I have a serialized avro binary data represented by byte[] where one of the
field is long with a logical type of TimeStamp.
Timestamp tsp = new Timestamp(1530228588182l);
Schema schema = SchemaBuilder.builder()
.record("hello")
.fields()
.name("
How does Kafka emulate exactly once processing currently? Does it require
the producer to send at least once and consumer to de dupe?
I did do my research but I feel like I am going all over the place so a
simple short answer would be great!
Thanks!
gt; crash, they can check the external store (HSFS, JSDC, etc) retrieve the
> last committed offset and seek the the next message and continue processing
> with no duplicates and exactly once semantics.
>
> -hans
>
>
>
>
> > On Dec 21, 2016, at 4:39 PM, kant kodali wrote:
Does the number of App instances and Zookeeper servers should be the same?
I understand the requirement of 2F+1 to tolerate F failures but this is to
tolerate failures of Zookeeper instances itself. But how about the number
of App instances ? For example say I have 3 zookeeper servers and I have 2
Hi All,
Can multiple Kafka consumers read from the same partition of same topic by
default? By default, I mean since group.id is not mandatory I am wondering
if I spawn multiple kafka consumers without specifying any group.id and
give them the same topic and partition name will they be able to rea
at 9:00 PM, Matthias J. Sax
wrote:
> Yes, you can do that.
>
> -Matthias
>
>
>
> On 3/30/17 6:09 PM, kant kodali wrote:
> > Hi All,
> >
> > Can multiple Kafka consumers read from the same partition of same topic
> by
> > default? By default, I mean
Hi All,
I have read enough blogs from Confluent and others and also books that
tried to talk about the differences between the two and while it is great
to know those differences I hardly find them any useful when it comes to
decision making process of which one to pick since I don't see the clear
Hi All,
1. I was wondering if anyone has seen or heard or able to create 1M or 10M
or 100M or 1B partitions in a topic? I understand lot of this depends on
filesystem limitations (we are using ext4) and the OS limitations but I
just would like to know what is the scale one had seen in production?
Forgot to mention: The question in this thread is for one node which has 8
CPU's 16GB RAM & 500GB hard disk space.
On Tue, May 16, 2017 at 2:06 AM, kant kodali wrote:
> Hi All,
>
> 1. I was wondering if anyone has seen or heard or able to create 1M or 10M
> or 100M or 1B
io/blog/how-to-choose-the-number-of-
> topicspartitions-in-a-kafka-cluster/
>
> -Sameer.
>
> On Tue, May 16, 2017 at 2:40 PM, kant kodali wrote:
>
> > Forgot to mention: The question in this thread is for one node which has
> 8
> > CPU's 16GB RAM & 500GB ha
What is your average message size and network speed?
On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
abhimanyunagr...@gmail.com> wrote:
> Hi Girish,
>
> I did not set any value for offsets.retention.minutes so therefore what I
> think is picking its default value i.e 1440 minutes so what do
2017 at 2:19 AM, kant kodali wrote:
> What is your average message size and network speed?
>
> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
> abhimanyunagr...@gmail.com> wrote:
>
>> Hi Girish,
>>
>> I did not set any value for offsets.retention.minut
@Abhimanyu You can try setting offset.retention = 30 (log.retention). At
most, you will have a storage overhead of 5 million msgs per day * 30
(days) * 8 bytes (for each offset) = 1.2GB (not that much since you have a
TB of hard disk)
On Sun, May 21, 2017 at 3:05 AM, kant kodali wrote
"Abhimanyu Nagrath"
> > wrote:
> >
> >> Hi Kant,
> >>
> >> Thanks for the suggestion.
> >>
> >>
> >> Regards,
> >> Abhimanyu
> >>
> >> On Sun, May 21, 2017 at 3:44 PM, kant kodali
> wrote:
> >>
> >
> ask whether this should also be tuned or not.
> >
> >
> > Regards,
> > Abhimanyu
> >
> >
> >
> >
> > On Mon, May 22, 2017 at 2:24 PM, kant kodali wrote:
> >
> >> @Abhimanyu Why do you think you need to set that? Did
54 matches
Mail list logo