Hello Apache Kafka community,
Current (Kafka 0.8.1.1) high-level API's KafkaConsumer is not lightweight
object, it's creation takes some time and resources, and it does not seem
to be thread-safe. It's API also does not support reuse, for consuming
messages from different consumer groups.
I see
Is there a supported C/C++ kafka client API?
Regards,
Virendra
Hi,
I might be a bit biased, but librdkafka is an established C/C++ client
library.
https://github.com/edenhill/librdkafka
Regards,
Magnus
2014-10-14 11:52 GMT+02:00 Virendra Pratap Singh
vpsi...@yahoo-inc.com.invalid:
Is there a supported C/C++ kafka client API?
Regards,
Virendra
Hi,
We’ve been using librdkafka for some time now in a semi-prod environment
without any issue so far.
That being said, that client is still not completely on par with the java one
in terms of functionality (zk support mostly)
Pascal
On Oct 14, 2014, at 09:45 AM, Magnus Edenhill
Stevo,
The new consumer API is planned for 0.9, not 0.8.2. You can take a look at
a detailed javadoc here
http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/org/apache/kafka/clients/consumer/KafkaConsumer.html
.
Can you explain why you would like to poll messages across
hey folks,
I have been testing a kafka cluster of 10 nodes on AWS using version
2.8.0-0.8.0
and see some behavior on failover that I want to make sure I understand.
Initially, I have a topic X with 30 partitions and a replication factor of
3. Looking at the partition 0:
partition: 0 - leader: 5
Could somebody help throw some light on why my commands might be hanging?
What's the easiest way to monitor and debug this problem?
On Mon, Oct 13, 2014 at 5:07 PM, Mohit Anchlia mohitanch...@gmail.com
wrote:
I am new to Kafka and I just installed Kafka. I am getting the following
error.
Magnus,
Do you have any plans to update the protocol to 0.9? I built a boost asio
based version half a year ago but that did only implement v0.8 and I have
not found time to upgrade it. It is a quite big job to have something equal
to java high and low level API.
/svante
Hi Svante,
yes, I will keep librdkafka in pace with the official Kafka
functionality/features/protocol changes as they are released upstream.
Next in the pipeline is broker based consumer groups to provide a proper
high levelish C/C++ consumer (and remedy the need for ZooKeeper).
Regards,
Magnus
Dear Kafka Users,
We are using kafka-2.8.0-0.8.8 in our application development. The
application is running with two brokers each having one partition.
We are using high level api to send message over async producer. While
testing through put of our application we have observed that more than
It's quite difficult to infer from the docs the exact techniques required
to ensure consistency and durability in Kafka. I propose that we add a doc
section detailing these techniques. I would be happy to help with this.
The basic question is this: assuming that I can afford to temporarily halt
I found this way:
zookeeper-shell.sh localhost:2181 get /config/topics/foo
{version:1,config:{cleanup.policy:compact}}
cZxid = 0x57
ctime = Tue Oct 14 12:02:54 PDT 2014
mZxid = 0x57
mtime = Tue Oct 14 12:02:54 PDT 2014
pZxid = 0x57
cversion = 0
dataVersion = 0
aclVersion = 0
A question about 0.8.1.1 and acks. I was under the impression that setting
acks to 2 will not throw an exception when there is only one node in ISR.
Am I incorrect ? Thus the need for min_isr.
On Tue, Oct 14, 2014 at 11:50 AM, Kyle Banker kyleban...@gmail.com wrote:
It's quite difficult to
I still have a question though. Is there a definitive way to tell if a
topic is configured for compaction? The way it seems to work now is that
the ZK config stores topic-specific config but the end results (whether
it's configured or not) is a combination of the topic-specific config and
the
Consider a 12-node Kafka cluster with a 200-parition topic having a
replication factor of 3. Let's assume, in addition, that we're running
Kafka v0.8.2, we've disabled unclean leader election, acks is -1, and
min.isr is 2.
Now suppose we lose 2 nodes. In this case, there's a good chance that 2/3
Wouldn't this work only for producers using random partitioning?
On Tue, Oct 14, 2014 at 1:51 PM, Kyle Banker kyleban...@gmail.com wrote:
Consider a 12-node Kafka cluster with a 200-parition topic having a
replication factor of 3. Let's assume, in addition, that we're running
Kafka v0.8.2,
Regarding (1), I am assuming that it is expected that brokers going down
will be brought back up soon. At which point, they will pick up from the
current leader and get back into the ISR. Am I right?
The broker will be added back to the ISR once it is restarted, but it never
goes out of the
The topic level configs override the broker defaults. So using the
kafka-topics tool should suffice.
On Tue, Oct 14, 2014 at 1:32 PM, Roger Hoover roger.hoo...@gmail.com
wrote:
I still have a question though. Is there a definitive way to tell if a
topic is configured for compaction? The way
It's not hanging. The console producer shell script allows you to enter
message via the console. So when you execute it, it waits for your console
input.
Also, the kafka distribution is missing a couple jar files:
slf4j-nop-1.7.2.jar
mx4j-tools.jar
You can google them and find them online.
Neha,
Thank you for the reply. It would be handy if a command line tool could
merge the two configs and show the effective result. Just an idea...
Thanks,
Roger
On Tue, Oct 14, 2014 at 2:29 PM, Neha Narkhede neha.narkh...@gmail.com
wrote:
The topic level configs override the broker
ack = 2 *will* throw an exception when there's only one node in ISR.
The problem with ack=2 is that if you have 3 replicas and you got acks
from 2 of them, the one replica which did not get the message can
still be in ISR and get elected as leader, leading for a loss of the
message. If you
The following is a bit weird. It indicates no leader for partition 4, which
is inconsistent with what describe-topic shows.
2014-10-13 19:02:32,611 WARN [main] kafka.producer.BrokerPartitionInfo:
Error while fetching metadata partition 4 leader: nonereplicas: 3
Also, which version of Kafka are you using?
Thanks,
Jun
On Tue, Oct 14, 2014 at 5:31 PM, Jun Rao jun...@gmail.com wrote:
The following is a bit weird. It indicates no leader for partition 4,
which is inconsistent with what describe-topic shows.
2014-10-13 19:02:32,611 WARN [main]
Hey Neha,
so I removed another broker like 30mn ago and since then basically the
Producer is dying with:
Event queue is full of unsent messages, could not send event:
KeyedMessage(my_topic,[B@1b71b7a6,[B@35fdd1e7)
kafka.common.QueueFullException: Event queue is full of unsent messages,
could not
Hi Chad,
Currently MM can only control which topics to MM from the source cluster,
and hence your two data centers share the same topics it currently does not
support that.
The usual practice for your purpose is to set up 4 Kafka clusters, 2 on
each data center (A and B), let's call them local
Hi Jun,
Thanks for responding...
I am using Kafka 2.9.2-0.8.1.1
I looked through the controller logs on a couple of nodes and did not find
any exceptions or error.
However in the state change log I see a bunch of the following exceptions -
[2014-10-13 14:39:12,475] TRACE Controller 3 epoch
26 matches
Mail list logo
