Thanks Guozhang for the pointer to the mapped NIO. The issue in my case was
related to the disk still being out of space (I thought I did free up some,
but I actually didn't). Curiously, I ran out of space on two occasions. In
one case the error message was clear No space left on device, and in
Hello everyone,
I have a few questions about the current status and future of the Kafka
consumers.
We have been working to adding Kafka support in Spring XD [1], currently
using the high level consumer via Spring Integration Kafka [2]. We are
working on adding features such as:
- the ability to
Hi all,
I have been trying out kafka benchmarks described in Jay's
benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machine.
I'm able to get similar results on a 4-node GbE
Hi,
I am trying to learn more on Kafka consuming patterns if we have partitions on
the topic and testing few scenarios. I would like to print the message when the
consumer consumes along with the partition id. How to get this done ?
We have simple consumer code (high level)
public void
I'm trying to understand the config options for auto-rebalancing. This
is what we have in /etc/kafka/server.properties for all the nodes:
auto.leader.rebalance.enable=true
leader.imbalance.per.broker.percentage=10
leader.imbalance.check.interval.seconds=300
We have 10 nodes for this topic
Hi everyone,
I'm looking for the API to create/manager topics. Is this class[1] the one
I should be using? Also what if I'm wanting to create the topic
synchronously?
[1] -
https://github.com/apache/kafka/blob/0.8.1.1/core/src/main/scala/kafka/admin/AdminUtils.scala#L149-L157
Bryan
Hey Manu,
I'm not aware of a benchmark on 10GbE. I'd love to see that though. Diving
into the results may help us find bottlenecks hidden by the slower network.
Can you figure out where the bottleneck is in your test? I assume this is a
single producer and consumer instance and you are using the
Hello Wes,
The document here is a bit misleading indeed:
http://kafka.apache.org/documentation.html#brokerconfigs
In Kafka a partition has a replica list {A,B,C..} and the first replica
would be the leader of the partition. When it is not the case, for example
since A is down B becomes the
The imbalance is measured wrt preferred leaders. i.e., for every
partition, the first replica in the assigned replica list (as shown in
the output of kafka-topics.sh) is called the preferred replica. On
each broker, the auto-balancer counts the number of partitions led by
that broker for which the
Inline..
On Tue, Nov 18, 2014 at 04:26:04AM -0500, Marius Bogoevici wrote:
Hello everyone,
I have a few questions about the current status and future of the Kafka
consumers.
We have been working to adding Kafka support in Spring XD [1], currently
using the high level consumer via Spring
Hi Joel,
Thanks for all the clarifications! Just another question on this: will
0.8.2 be backwards compatible with 0.8.1, just as 0.8.1 was with 0.8?
Generally speaking, would there be any concerns with using the 0.8.2
consumer with a 0.8.1 broker, for instance?
Marius
On Tue, Nov 18, 2014 at
Yes it should be backwards compatible. So for e.g., you should be able
to use an 0.8.1 client with an 0.8.2 broker. In general, you should
not upgrade your clients until after the brokers have been upgraded.
However, you can point an 0.8.2 client at an 0.8.1 broker. One wire
protocol change I'm
The new producer in 0.8.2 is considered stable, although it's relatively
new. Compared with the old producer, it has the following features.
1. Use non-blocking socket to send requests to the broker. So uses fewer
threads and have better throughput.
2. Bound the memory consumption.
3. Support a
Thanks Jay for the quick response.
Yes, it's a single producer and consumer both configured with multiple
threads but I'm not using the new producer.
CPU is typically 50% utilized on client and merely used on broker. Disks
aren't busy either as a lot of data are cached in memory.
Would you please
Hi Kakfa team,
So just monitor ZkClient-EventThread
http://zkclient-eventthread-65-dare-msgq00.sv.walmartlabs.com:9091/*
threads via ThreadInfo[] threads =
ManagementFactory.getThreadMXBean().; and if this ZkClient-EventThread
Not sure what happened, but the issue went away once revived the broker id
on a new host
But it does seem host D's ISR leadership could not be cleared until another
member of the ISR came back.somehow D was stale and remained stuck (and
clients therefore kept trying to connect to it)...
Hi Jun,
Is this the official java doc for the new producer (www.trieuvan.com)? I'm
not seeing any links to it (or any documentation) on the apache kafka site
(am I overlooking it)?
Should there be a link to it in the 0.8.2-beta documentation page?
Jason
On Tue, Nov 18, 2014 at 7:23 PM, Jun
0.8.2-beta java doc
https://archive.apache.org/dist/kafka/0.8.2-beta/java-doc/
/***
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
/
On
Yeah this will involve some experimentation.
The metrics are visible with jconsole or another jmx viewer.
It may also be worth looking at the cpu usage per-thread (e.g. start top
and press 't' I think).
Another simple test for broker vs client as the bottleneck is just to start
another producer
So that java-doc link includes a new KafkaConsumer (but it seems in other
threads that's not being referred to as ready for use until 0.9, is that
right?).Is there a way to know which parts of that javadoc are
considered beta-ready in 0.8.2 and which are not?
Jason
On Tue, Nov 18, 2014 at
Hey all,
We are working on a .net client, and I have a question about the OffsetRequest
api
(https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI)
It seems to indicate that you would always get the log end offset (tail
offset)
Try the following.
KAFKA_LOG4J_OPTS=-Dlog4j.configuration=file:/mylog4jdir
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
Thanks,
Jun
On Mon, Nov 17, 2014 at 11:11 AM, Jimmy John jj...@livefyre.com wrote:
How do I configure the application kafka log dir?
Right now
Either is fine. outgoing-byte-rate gives you the total and byte-rate
gives you per topic.
Thanks,
Jun
On Mon, Nov 17, 2014 at 3:58 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com
wrote:
Hi Jun,
So If I set the *metrics.sample.window.ms
http://metrics.sample.window.ms* to 1 minute, I will be
You need to define msg as MessageAndMetadatabyte[], byte[].
Thanks,
Jun
On Mon, Nov 17, 2014 at 10:43 PM, kade...@yahoo.com.INVALID
kade...@yahoo.com.invalid wrote:
Hi,
I am trying to learn more on Kafka consuming patterns if we have
partitions on the topic and testing few scenarios. I
Yes, that's the best option for now. The creation is async.
Thanks,
Jun
On Tue, Nov 18, 2014 at 8:55 AM, Bryan Baugher bjb...@gmail.com wrote:
Hi everyone,
I'm looking for the API to create/manager topics. Is this class[1] the one
I should be using? Also what if I'm wanting to create the
Is your ZK service alive at that point? If not, you just need to monitor
the ZK server properly.
Thanks,
Jun
On Mon, Nov 17, 2014 at 2:30 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com
wrote:
Hi Kafka Team,
I get following exception due to ZK/Network issues intermittently. How do
I
In the 0.8.2 final release, the java doc will be updated properly.
See KAFKA-1769
Thanks,
Jun
On Tue, Nov 18, 2014 at 8:46 PM, Jason Rosenberg j...@squareup.com wrote:
So that java-doc link includes a new KafkaConsumer (but it seems in other
threads that's not being referred to as ready for
Yes, we will preserve the last replica in ISR. This way, we know which
replica has all committed messages and can wait for it to come back as the
leader, if unclean leader election is disabled.
Thanks,
Jun
On Mon, Nov 17, 2014 at 11:06 AM, Jason Rosenberg j...@squareup.com wrote:
We have had
It is not possible, due to how the results for the offset request are stored
within the broker and API (as a map). You will have to do 2 requests to get
both offsets.
-Todd
On Nov 18, 2014, at 8:52 PM, Thunder Stumpges tstump...@ntent.com wrote:
Hey all,
We are working on a .net
Hi Jun,
ZK cluster are up and running. What is best way to programmatically
recover and I would try to exponential recovery process which I am willing
to implement.So do you think monitoring ZkClient-EventThread
http://zkclient-eventthread-65-dare-msgq00.sv.walmartlabs.com:9091/*
thread
Ok,
Makes sense. But if the node is not actually healthy (and underwent a hard
crash) it would likely not be able to avoid an 'unclean' restart.what
happens if unclean leader election is disabled, but there are no 'clean'
partitions available?
Jason
On Wed, Nov 19, 2014 at 12:40 AM, Jun
I feared that was the case. Well they were certainly right when they said this
API is slightly funky
Thanks for the quick confirmation.
Thunder
-Original Message-
From: Todd Palino [bonk...@gmail.com]
Received: Tuesday, 18 Nov 2014, 9:43PM
To: users@kafka.apache.org
32 matches
Mail list logo
