Hi Joel,
On Thu, Jun 4, 2015 at 8:52 PM, Joel Koshy wrote:
> Hi Otis,
>
> Yes this is a limitation in the old consumer. i.e., a number of
> per-topic/partition mbeans remain even on a rebalance. Those need to
> be de-registered. So if you stop consuming from some partition after a
> rebalance th
Hi Jason,
On Thu, Jun 4, 2015 at 5:55 PM, Jason Rosenberg wrote:
> what code do you use to poll the jmx end-point?
I used https://github.com/sematext/jmxc to get a "dump" of Consumer JMX.
It's a simple jar that you call with java -jar jmxc.jar PID > jmx.dump .
Otis
> On Thu, Jun 4, 2015 at
I just sent out a separate email about the project that I've been working
on, Burrow, to change the way we're monitoring consumer status.
Like Joel said, the mbeans can be a little tricky to work with. Internally
at LinkedIn, we've always depended more on using a tool like the Consumer
Offset Chec
I am very happy to introduce Burrow, an application to provide Kafka
consumer status as a service. Burrow is different than just a "lag checker":
* Multiple Kafka cluster support - Burrow supports any number of Kafka
clusters in a single instance. You can also run multiple copies of Burrow
in para
Hi Otis,
Yes this is a limitation in the old consumer. i.e., a number of
per-topic/partition mbeans remain even on a rebalance. Those need to
be de-registered. So if you stop consuming from some partition after a
rebalance that lag mbean currently remain which is why it remains
flat. This is a kn
On Thu, Jun 4, 2015 at 1:55 PM, Otis Gospodnetić wrote:
> Hi,
>
> On Thu, Jun 4, 2015 at 4:26 PM, Scott Reynolds
> wrote:
>
> > I believe the JMX metrics reflect the consumer PRIOR to committing
> offsets
> > to Kafka / Zookeeper. But when you query from the command line using the
> > kafka tool
what code do you use to poll the jmx end-point?
On Thu, Jun 4, 2015 at 4:55 PM, Otis Gospodnetić wrote:
> Hi,
>
> On Thu, Jun 4, 2015 at 4:26 PM, Scott Reynolds
> wrote:
>
> > I believe the JMX metrics reflect the consumer PRIOR to committing
> offsets
> > to Kafka / Zookeeper. But when you que
My apologies in advance - for cross-posting the question in both Dev and User
user-groups
Hi,
I have been using Kafka as DevOps user for a few months now and am now looking
at "building/testing" Kafka from source code.My gradle build is failing (source
from GitHub) with the error show below.Th
Hi,
On Thu, Jun 4, 2015 at 4:26 PM, Scott Reynolds wrote:
> I believe the JMX metrics reflect the consumer PRIOR to committing offsets
> to Kafka / Zookeeper. But when you query from the command line using the
> kafka tools, you are just getting the committed offsets.
>
Even if that were the ca
I believe the JMX metrics reflect the consumer PRIOR to committing offsets
to Kafka / Zookeeper. But when you query from the command line using the
kafka tools, you are just getting the committed offsets.
On Thu, Jun 4, 2015 at 1:23 PM, Otis Gospodnetic wrote:
> Hi,
>
> Here's something potentia
How's their garbage collection doing?
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Thu, Jun 4, 2015 at 12:43 AM, Anuja Pundlik (apundlik) <
apund...@cisco.com> wrote:
> Hi,
>
> I am using Kafka 0.8.2.1.
> W
Hi,
Here's something potentially useful.
1) Before: https://apps.sematext.com/spm-reports/s/eQ9WhLegW9 - the "flat
Lag situation"
2) I restarted the consumer whose lag is shown in the above graph
3) After restart: https://apps.sematext.com/spm-reports/s/4YGkcUP9ms - NO
lag at all!?
So that 815
Hi Jason,
(note: Kafka 0.8.2. GA)
Got some new info below! Could be a Kafka metrics bug
On Thu, Jun 4, 2015 at 2:11 PM, Jason Rosenberg wrote:
> I assume you are looking at a 'MaxLag' metric, which reports the worst case
> lag over a set of partitions.
No, we're looking at MBeans that lo
I assume you are looking at a 'MaxLag' metric, which reports the worst case
lag over a set of partitions. Are you consuming multiple partitions, and
maybe one of them is stuck?
On Tue, Jun 2, 2015 at 4:00 PM, Otis Gospodnetic wrote:
> Hi,
>
> I've noticed that when we restart our Kafka consumer
Sebastien, I think you may have an off by 1 error (e.g. batch should be
0-199, not 1-200). Thus you are sending 2 batches each time (one for 0,
another for 1-199).
Jason
On Thu, Jun 4, 2015 at 1:33 PM, Jiangjie Qin
wrote:
> From the code you pasted, that is old producer.
> The new producer cla
>From the code you pasted, that is old producer.
The new producer class is org.apache.kafka.clients.producer.KafkaProducer.
The new producer does not have sticky partition behavior. The default
partitioner use round-robin like way to send non-keyed messages to
partitions.
Jiangjie (Becket) Qin
O
Hi,
I am using Kafka 0.8.2.1.
We have 1 zookeeper, 3 kafka brokers.
We have 9 topics, out of which 1 topic has 18 partitions, while another has 12
partitions. All other topics have 1 partition each.
We see that idle kafka brokers (not carrying any message) are using more than
50% of CPU. See to
17 matches
Mail list logo
