Luckily, I was just reviewing a lot of this information for my ApacheCon
talk next week. Those slides, and the video (I hope) will be published as
soon as the talk is done. I'll give you the information I have from
LinkedIn's point of view, but out of order :)
Our Kafka brokers are all the same
Hi Mayuresh,
We are having only one consumer in the group and only one partition for
that topic.
We have set auto.commit.enable false, zookeeper.session.timeout.ms=6,
rebalance.backoff.ms to 2000 and rebalance.max.retries to 20.
Thanks!
On Mon, Apr 6, 2015 at 9:59 PM, Mayuresh Gharat
Hey Liang,
Have you looked at the quick start here:
https://kafka.apache.org/documentation.html#quickstart
In Kafka, on the producer side, there is no concept of ³commit². If you
are producing using KafkaProducer, you can do a send.get(), this is a
synchronized send so if no exception was
Also if you are using Kafka from the latest trunk, KafkaProducer has a
flush() interface that you may call. This will ensure all the message
previously sent from send() methods are sent to Kafka server.
On 4/3/15, 3:38 PM, Sun, Joey joey@emc.com wrote:
Hello, group
I am a newbie to Kafka. I
I have had some brokers die because of lack of disk space. The logs for all
partitions were way higher (5G+) than I would have expected given the how I
configured them for (100 MB size AND 1h rollover).
What is the recommended way of recovering from this error. Should I delete
certain log files
We're showing a constant level of unclean leader election errors. I'd
like to investigate but I'm not quite sure how to approach it. Is there
a doc somewhere that goes into some detail on what to look at?
Thanks,
Wes
I answered this in IRC, but the issue is that retention depends on the
modification time of the log segments on disk. When you copy a partition
from one broker to another, the mtime of the log segments on the new broker
will be now. That means the retention clock starts over again. This means
that
Hi Sandeep / Alec,
I have been testing on trunk's KafkaConsumer with partition subscription
only, and it works for me. The topic subscription does not work yet as we
are still implementing that feature.
Could you confirm if you are using trunk's KafkaConsumer and subscribe to
partitions?
Thanks for your info, Becket.
Does it mean I should program for it? is there any other app can gracefully
glue access_log to Kafka's producer?
Thanks
Liang
-Original Message-
From: Jiangjie Qin [mailto:j...@linkedin.com.INVALID]
Sent: Monday, April 06, 2015 11:46 AM
To:
Jack,
Your theory is correct if your consumer config set auto.offset.reset to
latest and you do not have any committed offsets before. Could you list
your consumer configs and see if that is the case?
Guozhang
On Mon, Apr 6, 2015 at 3:15 PM, Jack jac...@gmail.com wrote:
Hi folks,
I have a
Hi Guozhang,
Thanks so much for replying, first of all.
Here is the config we have:
group.id - 'some unique id'
zookeeper.connect - 'zookeeper host'
auto.commit.enabled - false
'auto.offset.reset' - largest
consumer.timeout.ms - -1
fetch.message.max.bytes - 10M
So it seems like we need to make
Jack,
Could you just change auto.offset.reset to smallest and see if this issue
goes away? It is not related to the producer end.
Guozhang
On Mon, Apr 6, 2015 at 4:14 PM, Jack jac...@gmail.com wrote:
Hi Guozhang,
Thanks so much for replying, first of all.
Here is the config we have:
Hi folks,
I have a quick question.
We are using 0.8.1 and running into this weird problem. We are using
HighLevelConsumer for this topic. We created 64 partitions for this
message.
In our service, we first create a Consumer object as usual, and then we
went ahead, calls 'createMessageStreans'
What is the best practice for reporting the lag on individual consumer
groups (e.g., to Graphite)?
A recent form post (http://search-hadoop.com/m/4TaT4x9qWm1) seems to
indicate that parsing the output of the consumer offset checker tool and
reporting that independently is what folks do. Is there
Hi all,
I have collected our logs and paired them down. Jiangjie, I sent you more
complete logs directly to your mailto from the list (minus the .INVALID).
Please let me know if you did not get them. I cannot post them here in public.
For others, and to continue the conversation, during the
Did you turn on automatic offset committing? If yes then this issue should
not happen as later runs will just consume data from the last committed
offset.
Guozhang
On Mon, Apr 6, 2015 at 5:16 PM, Jack jac...@gmail.com wrote:
Hi Guozhang,
When I switched to auto.offset.reset to smallest, it
Hi Guozhang,
When I switched to auto.offset.reset to smallest, it will work. However, it
will generate a lot of data and it will slow down the verification.
Thanks,
-Jack
On Mon, Apr 6, 2015 at 5:07 PM, Guozhang Wang wangg...@gmail.com wrote:
Jack,
Could you just change auto.offset.reset
Hi,
We track our Consumer Lag and all other Kafka metrics with SPM
http://sematext.com/spm whose agent collects this info from Kafka
Consumers. Here's a chart of that:
https://apps.sematext.com/spm-reports/s/mcBHvsXwgj -- this public chart
doesn't show all filtering option, but you can filter
Hi,
These days I have been focus on Kafka 0.8 replication design and found
three replication design proposals from the wiki (according to the document,
the V3 version is used in Kafka 0.8 release).
But the v3 proposal is not complete and is inconsistent with the
release.
Is
How about the first run then? If we use largest as auto.offset.reset
value, what value will these consumers get? I assume it will point to the
latest position in the log. Is that true? Just you know, we can't have a
warm up run so that the later runs can use the committed offset by that run.
To
Hi All,
I tried lot to unsubscribe from this list, even I send several e-mails. Please
unsubscribe from this list, since mailbox is getting filled.
Thanks,
pradeep
-Original Message-
From: Rajiv Kurian [mailto:ra...@signalfuse.com]
Sent: Tuesday, April 07, 2015 12:40 AM
To:
@Pradeep,
On Tue, Apr 7, 2015 at 9:30 AM, pradeep.si...@wipro.com wrote:
I tried lot to unsubscribe from this list, even I send several e-mails.
Please unsubscribe from this list, since mailbox is getting filled.
See here: http://kafka.apache.org/contact.html for instructions on how
to
Sorry sent early by mistake.
I tried this command
/usr/share/fk-3p-kafka/bin/kafka-topics.sh --zookeeper zookeeper_url
--alter --topic benchmark-test-2
And it returned without doing anything. I was actually trying to see if
there is a way to alter all topics as once and change the replication
Hi All,
We are facing issue with Kafka high Level consumers. We are using 0.8.1.1
version. Sometimes after restart consumers picks the messages sometimes it
starts but not receives any messages. Is high level consumer is not
reliable?
I have checked with the log. Even if re-balance succeed,
Hi,
These days I have been focus on Kafka 0.8 replication design and found
three replication design proposals from the wiki (according to the document,
the V3 version is used in Kafka 0.8 release).
But the v3 proposal is not complete and is inconsistent with the
release.
Is
Yes, the wiki is a bit old. You can find out more about replication in the
following links.
http://kafka.apache.org/documentation.html#replication
http://www.slideshare.net/junrao/kafka-replication-apachecon2013
#1, #2, #8. See the ZK layout in
It actually depends on how many consumers you have in the same group and
how many partitions the particular topic has.
Can you elaborate on your configuration?
Thanks,
Mayuresh
On Mon, Apr 6, 2015 at 3:35 AM, Madhukar Bharti bhartimadhu...@gmail.com
wrote:
Hi All,
We are facing issue with
Hello,
I am trying to understand some of the common Kafka deployment sizes
(small, medium, large) and configuration to come up with a set of common
templates for deployment on Linux. Some of the Qs to answer are:
- Number of nodes in the cluster
- Machine Specs (cpu, memory, number
28 matches
Mail list logo
