Hi,
it's probably beyond that. it may be an issue with the number of files
Kafka can have opened concurrently.
A previous conversation with Joe about (build failes for latest stable
source tgz (kafka_2.9.2-0.8.1.1)) turned out to discuss this (Q's by Joe,
A's by me):
1. what else on the logs?
Hi folks,
One of things stopping us from upgrading version of Kafka from 0.7 to 0.8
is the absence of a native PHP client. A search points to this:
https://github.com/salebab/phpkafka. Does anyone have experience with it ?
We require a PHP client due to high throughput, low latency existing
(2)
configure enough retries + backoff time in the producer (so that new
leaders can be elected during failure).
Ya, I’ve been doing this, I need to get it just right.
(1) use ack=-1 in the producer;
We use ack=1. If I configure (2) properly, I shouldn’t need ack=-1, will I? I
don’t think
Hi,
We plan to capture various metrics exposed in Kafka via JMX and wanted to
check if anybody has a plugin file which could be reused.
We found one such plugin file here
https://github.com/deathsyn/chef-kafka/blob/master/templates/default/collectd_kafka-broker.conf.erb
but
it seems to be
We use jmxtrans to pull data out of JMX.
https://github.com/wikimedia/puppet-kafka/blob/master/kafka-jmxtrans.json.md
On Sep 9, 2014, at 7:54 AM, Navneet Gupta (Tech - BLR)
navneet.gu...@flipkart.com wrote:
Hi,
We plan to capture various metrics exposed in Kafka via JMX and wanted to
For your first question, I don't know of a way to do that.
For the second question, consumers commit the offset after every/a few
reads. When the consumer restarts, it starts from the last committed offset.
On Tue, Sep 9, 2014 at 7:23 PM, siddharth ubale siddharth.ub...@gmail.com
wrote:
Hi
looks like I can't edit the wiki page.
feel free to add https://github.com/airbnb/kafka-statsd-metrics2 to the
page.
thanks
On Tue, Sep 9, 2014 at 8:10 AM, Andrew Otto ao...@wikimedia.org wrote:
We use jmxtrans to pull data out of JMX.
noticed one important thing. topic foo's partition 1 and 2 have empty .log
file on replicas. I suspect replication doesn't create the partition dir on
broker 0 in this case, which then cause the WARN logs.
On Mon, Sep 8, 2014 at 11:11 PM, Steven Wu stevenz...@gmail.com wrote:
sorry. forgot to
Hello Kafka users,
Meet Bruce, a producer daemon developed at Tagged, Inc.
(http://www.tagged.com). We are open sourcing Bruce because we have
found it useful at Tagged, and believe others may also benefit from it.
Bruce is available on GitHub (https://github.com/tagged/bruce ).
We developed
This is very cool, Dave. Thanks to you and your team for this work. This
may save us some work in the future.
Special thanks for also paying such attention to monitoring!
On Tue, Sep 9, 2014 at 12:24 PM, Dave Peterson dspeter...@tagged.com
wrote:
Hello Kafka users,
Meet Bruce, a producer
Very cool, can you update the Wiki please?
https://cwiki.apache.org/confluence/display/KAFKA/Clients maybe under a
section called Daemon? Or something?
/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
HI Jun,
Thanks for clarification. Follow up questions, does new producer solve the
issues highlight. In event of compression and async mode in new producer,
will it break down messages to this UPPER limit and submit or new producer
strictly honor batch size. I am just asking if compression
Kafka Team,
Can you please let me know what each of following Metrics means ? Some of
them are obvious, but some are hard to understand. My Topic name is
*TOPIC_NAME*.
can we enforce a Topic Name Convention or Metric Name Convention. Because
in previous version of Kafka, we have similar
Hmm, that seems to indicate that the leader info is not propagated properly
from the controller to broker 0. In the state-change log of broker 0, do
you see anything related to these 2 partitions when broker 0 comes up?
Thanks,
Jun
On Tue, Sep 9, 2014 at 9:41 AM, Steven Wu stevenz...@gmail.com
Are you starting the broker in some container? You want to make sure that
the container doesn't overwrite the open file handler limit.
Thanks,
Jun
On Tue, Sep 9, 2014 at 12:05 AM, Shlomi Hazan shl...@viber.com wrote:
Hi,
it's probably beyond that. it may be an issue with the number of files
nope. sate-change log files only had some warnings regarding other
partitions. nothing related to these two partitions.
2014-09-09 02:54:30,579 WARN kafka.utils.Logging$class:83
[kafka-request-handler-1] [warn] Broker 0 received invalid LeaderAndIsr
request with correlation id 497 from
No, the new producer doesn't address that problem.
Thanks,
Jun
On Tue, Sep 9, 2014 at 12:59 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com
wrote:
HI Jun,
Thanks for clarification. Follow up questions, does new producer solve the
issues highlight. In event of compression and async mode in
assuming you are talking about controller log on broker 0, there is nothing
there.
$ less kafka-controller
2014-09-09 01:15:02,600 INFO kafka.utils.Logging$class:68 [main] [info]
[ControllerEpochListener on 0]: Initialized controller epoch to 5 and zk
version 4
2014-09-09 01:15:02,655 INFO
ah. maybe you mean the controller log on leader/controller broker 5. yes. I
do noticed some errors regarding these two partitions.
[2014-09-09 01:10:53,651] ERROR Controller 5 epoch 5 encountered error
while changing partition [foo,2]'s state from New to Online since
LeaderAndIsr path alrea
dy
Hi Jun,
Is there any plug-ability that Developer can customize batching logic or
inject custom code for this ? Shall I file Jira for this issues.
Thanks,
Bhavesh
On Tue, Sep 9, 2014 at 3:52 PM, Jun Rao jun...@gmail.com wrote:
No, the new producer doesn't address that problem.
Thanks,
Jun
Hi,
I have topic name with “.” So, it is hard to distinguish metric name and
topic
If your topics or hosts have dashes or underscores or your topic has dots
you will have trouble extracting things from JMX. We provided a patch to
change delimiter character in MBeans and are hoping it gets
Thanks, I was using without JMX. I will go through doc. But how about
Topic or Metric name Topic Name Convention or Metric Name Convention ? The
dot notation with topic having a .. Any future plan to enforce some
stand rules.
Thanks,
Bhavesh
On Tue, Sep 9, 2014 at 3:38 PM, Jay Kreps
previous email is from state-change.log. also found this WARN in
controller.log
[2014-09-09 01:10:53,669] WARN [Controller 5]: Cannot remove replica 0 from
ISR of partition [cdn_selection_runtime_data,8] since it is not in the ISR.
Leader = 1 ; ISR = List(1) (kafka.controller.KafkaController)
Hi,
Thanks Sharninder!!
Does everyone else share the same view as sharninder that it is not
possible to store data in database and we can only store data in configured
folder at consumer end .please do let me know your views.
Thanks,
Siddharth
On Tue, Sep 9, 2014 at 9:10 PM, Sharninder
24 matches
Mail list logo
