回复：How Kafka work internally?

you don’t need the full broker list for “metadata.broker.list”, one healthy broker will get metadata from zk, and then return to the produce client. 原始邮件 发件人:Kiran singhkim.sing...@gmail.com 收件人:usersus...@kafka.apache.org 发送时间:2015年10月19日(周一) 19:43 主题:How Kafka work internally? Hi I am using

How Kafka work internally?

Hi I am using producer API to send data to a topic. For this I am using "metadata.broker.list" to specify my broker list and i am able to send data successfully. But i am looking for how Kafka server handle a producer API means 1. which broker act as a coordinator for the send request. As we

Re: How Kafka work internally?

What mean by healthy broker? On Mon, Oct 19, 2015 at 5:26 PM, 黄刚 wrote: > you don’t need the full broker list for “metadata.broker.list”, one > healthy broker will get metadata from zk, and then return to the produce > client. > > > 原始邮件 > 发件人:Kiran

Re: How Kafka work internally?

The broker is working normally. 原始邮件 发件人:Kiran singhkim.sing...@gmail.com 收件人:usersus...@kafka.apache.org 发送时间:2015年10月19日(周一) 20:00 主题:Re: How Kafka work internally? What mean by healthy broker? On Mon, Oct 19, 2015 at 5:26 PM, 黄刚 gang.hu...@dmall.com wrote: you don’t need the full broker

Spamming kafka mailing list

Kiran, Please stop spamming the mailing list. There is enough Kafka documentation for basic queries. Please spend time finding the answers before sending queries to Kafka mailing list. In case you still not find answers Kafka community is there to help. Genuine queries do not get proper attention

Re: Synchronous producer performance is very low

The bottom line is synchronous will never be as fast as asynchronous. When using asynchronous mode you get the benefits of batching, compression, reduced network roundtrips, etc. You do loose the ability to achieve absolute ordering, but do you really need that? Can you design you system not to?

Kafka topic message consumer fetch response time checking

If one wanted to check response time for a consumer fetching a topic message (on the client side), similar to checking an HTTP request's response time for the web, what's the best approach to take? I notice the kafka shell scripts if used for that have some startup overhead if used to assess

kafka.network.RequestMetrics.FetchFollower metrics

I’m comparing some of the performance differences between two clusters that are theoretically identical except for the physical networks they’re on (configs and hardware are identical, each has a local zookeeper ensemble). I noticed that the slower cluster is exhibiting significantly higher

Re: Client consumer question

Hi Mohit, Are you referring to the new Java consumer or the old consumer? Or more specifically what examples doc are you referring to? Guozhang On Mon, Oct 19, 2015 at 10:01 AM, Mohit Anchlia wrote: > I see most of the consumer examples create a while/for loop and then

Re: Client consumer question

By old consumer you mean version < .8? Here are the links: https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example On Mon, Oct 19, 2015 at 12:52 PM, Guozhang Wang wrote: > Hi

Re: Spamming kafka mailing list

Hi Kiran, Your enthusiasm for Kafka has been noticed. Here are a few resources you can use to help answer your questions and improve your Kafka knowledge: - Kafka Documentation - Kafka Development Wiki

Re: Client consumer question

The links you are referring are for the old consumer. If you are using the ZooKeeper based high-level version of the old consumer which is described in the second link, then failures are handled and abstracted from you so that if there is a failure in the current process, its fetching partitions

Re: Client consumer question

Is 0.9.0 still under development? I don't see it here: http://kafka.apache.org/downloads.html On Mon, Oct 19, 2015 at 4:05 PM, Guozhang Wang wrote: > The links you are referring are for the old consumer. > > If you are using the ZooKeeper based high-level version of the old

Re: Kafka 8.2.2 doesn't want to compress in snappy

You will need to upgrade the broker to 0.8.2.2. Broker currently recompresses messages. In 0.8.2.1, the snappy jar has a bug that causes data explosion. We fixed the snappy jar in 0.8.2.2. If you upgrade the broker to 0.8.2.2, it will pick up the fixed snappy jar. Thanks, Jun On Sat, Oct 17,

Re: Client consumer question

It is not released yet, we are shooting for Nov. for 0.9.0. Guozhang On Mon, Oct 19, 2015 at 4:08 PM, Mohit Anchlia wrote: > Is 0.9.0 still under development? I don't see it here: > http://kafka.apache.org/downloads.html > > On Mon, Oct 19, 2015 at 4:05 PM, Guozhang

Multithreading to improve producer performance.

Hi All, I'm new to Kafka and in the process of writing a producer. Just to give you the context, my producer reads a binary file, decodes it according to a predefined structure (message length followed by the message) and publishes the decoded messages based on its type to the topic. For instance

Regarding Custom Partitioner

Hello I created Custom partitioner for my need implemented Partitioner interface Override this method public int partition(Object key, int a_numPartitions){ return partitionId; } We have something called as We are using key as correlationId, That will be unique for each message .

Re: Getting error while reading message

Are you consuming data in kafkaStream with byte[] argument in generics On Sun, Oct 18, 2015 at 4:20 PM, Kiran Singh wrote: > Hi Pratapi > > I am using following serializer property at producer side: > > ("serializer.class", "kafka.serializer.StringEncoder"); > > And at

Re: Synchronous producer performance is very low

Hi Grant, Thanks for the reply. Yes I meant data is received on the other end in 1sec while using asynchronous producer. We are using the new producer and absolute ordering is our prime concern too. Thanks, Santhakumari On Mon, Oct 19, 2015 at 9:12 PM, Grant Henke wrote: >

It's 5.41am, we're after 20+ hours of debugging our prod cluster. See NotAssignedReplicaException and UnknownException errors. Help?

Hi! We're running a 5-machine production Kafka cluster on version 0.8.1.1. Yesterday we had some disk problems on one of the replicas and decided to replace that node with a clean one. That's when we started experiencing many different problems: - partition replicas are still assigned to the old

RE: How Kafka work internally?

But how can I know which broker is working fine at run time. Anyways but I am looking how API flow at Kafka side. Can you please share any link which explain this. Thanks for your response. -Original Message- From: "黄刚" Sent: ‎19-‎10-‎2015 05:48 PM To: "users"

Client consumer question

I see most of the consumer examples create a while/for loop and then fetch messages iteratively. Is that the only way by which clients can consumer messages? If this is the preferred way then how do you deal with failures, exceptions such that messages are not lost. Also, please point me to