Re: Where is replication factor stored?

> On Oct 16, 2015, at 1:19 PM, Guozhang Wang wrote: > > Replication factor is stored as topic configs that are introduced since > 0.8.1, you can find it in the wiki you mentioned. > Ah, I didn't notice the /config section. But it still doesn't show the replication factor.

Re: Where is replication factor stored?

The replica list can be from at /brokers/topics// partitions//state Guozhang On Fri, Oct 16, 2015 at 2:06 PM, Edward Ribeiro wrote: > Umm... the replica *assignment* gets stored under /brokers/topics/ > as "partitions" field, no? Under /brokers/topics//partitions

Where is replication factor stored?

Hi, Where is the replication factor for a topic stored? It isn't listed at https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper. But the kafka-topics --describe command returns something. Where is it finding that? Thanks, -James

Re: Where is replication factor stored?

We don't store the replication factor per-se. When the topic is created, we use the replication factor to generate replica-assignment, and the replica assignment gets stored in ZK under: /brokers/topics//partitions/... This is what gets modified when we re-assign replicas. Hope this helps. Gwen

Re: Where is replication factor stored?

Umm... the replica *assignment* gets stored under /brokers/topics/ as "partitions" field, no? Under /brokers/topics//partitions there is a znode for each partition number with a 'state' as a sub-znode, right? James, by doing via zkCli.sh: get /brokers/topics/ You get a json like below:

Re: Where is replication factor stored?

Actually, be very careful with this. There are two different things stored in Zookeeper, and depending on what you're interested in you want to make sure you're looking at the right one. If you want to know the replica assignment - that is, what brokers a given partition is assigned to - you need

Re: Where is replication factor stored?

Gwen is right, I was wrong :P kafka-topics --describe reads the ZK path that Gwen mentioned to return the replica list, which gives you the idea about replication factor as well. Thanks, Guozhang On Fri, Oct 16, 2015 at 1:51 PM, Gwen Shapira wrote: > We don't store the

Re: Where is replication factor stored?

Umm... Reading the TopicCommand code https://github.com/apache/kafka/blob/362613347371e9d74184e900ab80ba230940a5c8/core/src/main/scala/kafka/admin/TopicCommand.scala#L192 , it looks like the replication factor (for --describe option, at least) is calculated by: 1) retrieving the "partitions" map

Re: Where is replication factor stored?

Hey, Guozhang, On Fri, Oct 16, 2015 at 6:20 PM, Guozhang Wang wrote: > The replica list can be from at /brokers/topics// > partitions//state > Nice, good to know. Thanks! :) Regards, Edward​

Re: Where is replication factor stored?

Sorry, I forgot the tl;dr on that :) If you want to know the replication factor for a given partition, you want to check the length of the replica list in the /brokers/topic/(topic) data for that partition. Note that all the partitions for a topic do not have to have the same replication factor

Re: Where is replication factor stored?

Replication factor is stored as topic configs that are introduced since 0.8.1, you can find it in the wiki you mentioned. Guozhang On Fri, Oct 16, 2015 at 12:33 PM, James Cheng wrote: > Hi, > > Where is the replication factor for a topic stored? It isn't listed at >

Re: ("offsets.storage","kafka") not working with Consumer API

Hello Kiran, Check how many brokers you have in the cluster. Consumer offsets topic requires by default at least 3. In dev environment you could lower replication factor for that topic (see broker config options). Kind regards, Stevo Slavic. On Fri, Oct 16, 2015, 07:31 Kiran Singh

Kafka Synchronous Performance issue

Hi All, We are using Kafka-Net(C# SDK) for our kafka messaging system, we tested Producer to produce message to Kafka with Synchronous and Asynchronous mode. But In Asynchronous mode it performs well than Synchronous. Synchronous mode takes 2 minutes to produce 1000 message where as Asynchronous

Re: ("offsets.storage","kafka") not working with Consumer API

Hi Stevo I have 3 broker in my kafka cluster. On Fri, Oct 16, 2015 at 12:21 PM, Stevo Slavić wrote: > Hello Kiran, > > Check how many brokers you have in the cluster. Consumer offsets topic > requires by default at least 3. In dev environment you could lower > replication

Re: Getting error code 15 while connect to the offset manager.

Found following error in server.log file under log folder. ERROR Closing socket for /x.x.x.x because of error (kafka.network.Processor) java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at

Re: Getting error code 15 while connect to the offset manager.

Hi Mayuresh Can you please share how and from where i can collect logs. On Fri, Oct 16, 2015 at 9:53 AM, Kiran Singh wrote: > Hi Mayuresh > > I am following link > > https://cwiki.apache.org/confluence/display/KAFKA/Committing+and+fetching+consumer+offsets+in+Kafka > >

Re: How to get group list in kafka?

By when we can get 0.9.0.0 version of kafka? On Fri, Oct 16, 2015 at 10:07 AM, Kiran Singh wrote: > Hi Ashish > > I am using kafka_2.10-0.8.2.2.jar > > On Fri, Oct 16, 2015 at 9:25 AM, Ashish Singh wrote: > >> Hello Kiran, >> >>

Spring xd stream - kafka

Hi Kafka Users I am new to streams in spring xd and trying to consumer data from a topic. I downloaded the spring xd single node on my laptop and ran the following command: 1. stream create myKafka --definition "kafka --zkconnect=localhost:2181 --topic=mytopic | log" --deploy I did not see the

kafka consumer shell scripts (and kafkacat) with respect to JSON pretty printing

I was wondering, do the kafka consumer shell scripts (high and low level ones) and kafkacat do any pre-processing of the topic messages before outputting to stdout or does it just output "as is" in the format the message originally came in through kafka from the producer? Meaning pretty printed