Re: zookeeper upgrade or remove zookeeper dependency

Hi Mingtao, We are shooting to cut the 0.8.2 branch this month. Guozhang On Thu, Sep 18, 2014 at 10:36 AM, Mingtao Zhang wrote: > Good to know. Does it mean release will go out after those bug is fixed or > moved to newer release? :) > > Best Regards, > Mingtao > > On Wed, Sep 17, 2014 at 9:34

Re: Handling errors in the new (0.8.2) Java Client's Producer

Hello Andrew, I think you would want a sync producer for your use case? You can try to call get() on the returned metadata future of the send() call instead of using a callback; the pattern is something like: for (message in messages) producer.send(message).get() The get() call will block un

Re: How to use RPC mechanism in Kafka?

Do you mean that you want to know the protocol? https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol Guozhang On Fri, Sep 19, 2014 at 12:18 PM, lavish goel wrote: > Hi, > > Please tell me how to use request/response mechanism in kafka? > Thanks > Lavish Goel > --

Re: How to use RPC mechanism in Kafka?

> > > > > > Sorry for again asking the question. > > > > > > > > > > > > I want to know, Is it possible: > > > > > > > > > > > > Let say, I have producer PA,PB,PC. They send request messages > A,B,C &

Re: Questions about Kafka 0.9 API changes

Hello, 1) The new consumer clients will be developed under a new directory. The old consumer, including the SimpleConsumer will not be changed, though it will be retired in the 0.9 release. 2) I am not very familiar with HTTP wrapper on the clients, could someone who have done so comment here? 3

Re: Questions about Kafka 0.9 API changes

Hello, For your use case, with the new consumer you can still create a new consumer instance for each topic / partition, and remember the mapping of topic / partition => consumer. The upon receiving the http request you can then decide which consumer to use. Since the new consumer is single threa

Re: Questions about Kafka 0.9 API changes

er API, if there are use cases which just work much better > using it. > > Greetings > Valentin > > On 23/09/14 18:16, Guozhang Wang wrote: > > Hello, > > > > For your use case, with the new consumer you can still create a new > > consumer instance for each

Re: Producer errors (failed to send producer request, failed to send requests for topics)

Kyle, the new producer will handle this exception with the specific exception type, and the callback handling function can treat it accordingly. Could you give the new producer client a try and see if that is better handled now? On Tue, Sep 23, 2014 at 8:30 PM, Kyle Banker wrote: > Thanks so muc

Re: Log.scala

Hi Manjunath, This check "!appendInfo.offsetsMonotonic" makes sure that there will be no gaps. And the other part "appendInfo.firstOffset < nextOffset.get" I agree that it seems redundant. Guozhang On Fri, Sep 26, 2014 at 5:52 AM, Manjunath Shivakumar < manjunath.shivaku...@betfair.com> wrote:

Re: Questions about Kafka 0.9 API changes

ection pool > >> holds > >> >> upto 10 SimpleConsumer connections. So all in all I get a maximum of > >> >> 50 > >> >> open connections per web application server. And with that I am able > >> >> to > >> >> handle mo

Re: Re: Log.scala

gt; But doesn't this check "!appendInfo.offsetsMonotonic" only ensure there > are no gaps > in the given message set. How do we ensure there are no gaps between this > message set > and previous logged offsets. > > Thanks, > Manju > > -- Forwarded mess

Re: rebalance brokers

Hi Guangle, Replication is only introduced in 0.8, with 0.7 there should not have leader / follower replicas. Could you verify the version of your Kafka cluster? Guozhang On Mon, Sep 29, 2014 at 4:30 PM, Guangle Fan wrote: > Hi, All > > We have some old kafka nodes that are still running 0.7.

Re: conflicted ephemeral node error

Hi Snehalata, Did you see this log only a few times or it keep spilling in the log file? Guozhang On Sun, Sep 28, 2014 at 11:20 PM, Snehalata Nagaje < snehalata.nag...@harbingergroup.com> wrote: > > > Hi , > > > I am getting this error in kafka logs > > I wrote this conflicted ephemeral node >

Re: multi-node and multi-broker kafka cluster setup

Hello, In general it is not required to have the kafka brokers installed on the same nodes of the zk servers, and each node can host multiple kafka brokers: you just need to make sure they do not share the same port and the same data dir. Guozhang On Mon, Sep 29, 2014 at 8:31 PM, Sa Li wrote:

Re: rebalance brokers

ept of replicas on it. Is there a way to rebalance > partitions across all brokers when adding new nodes ? > > On Mon, Sep 29, 2014 at 6:11 PM, Guozhang Wang wrote: > > > Hi Guangle, > > > > Replication is only introduced in 0.8, with 0.7 there should not have >

Re: Created topic by 2 partitions, only can use the one partition

Hi Jiang, Which producer client did you use? And did you specify any keys for your sent messages? Guozhang On Tue, Sep 30, 2014 at 10:45 AM, Jiang Jacky wrote: > Hi, Guys > It is very weird, I created a topic with 2 partitions couple weeks ago, and > I can only production the message to partit

Re: kafka producer performance test

Hello Sa, KAFKA-1490 introduces a new step of downloading the wrapper, details are included in the latest README file. Guozhang On Thu, Oct 2, 2014 at 11:00 AM, Sa Li wrote: > Thanks, Jay, > > Here is what I did this morning, I git clone the latest version of kafka > from git, (I am currently

Re: Different free space and log files between nodes

Hello Dayo, This is a known issue, since today Kafka's log rolling / cleaning policy depends on the creation timestamp of the segment files, which could be modified upon partition migration / broker restart, it can cause the server to not honor the specified log cleaning config. Some more details

Re: kafka producer performance test

get more log output. > > BUILD FAILED > > > ---- > > Seems it is really not that straightforward to build > > > thanks > > > > > On Thu, Oct 2, 2014 at 12:56 PM, Guozhang Wang wrote: > > > Hello Sa, > > > > KAFK

Re: create topic in multiple node kafka cluster

Sa, Usually you would not want to set up kafka brokers at the same machines with zk nodes, as that will add depending failures to the server cluster. Back to your original question, it seems your zk nodes do not form an ensemble, since otherwise their zk data should be the same. Guozhang On Thu

Re: mount of messages produced and those consumed are not equal

Hi Shuo, What are your producer and consumer config values? Guozhang On Fri, Oct 10, 2014 at 3:47 AM, Shuo Chen wrote: > Dear folks, >I have run some basic tests using kafka. I have started 7 brokers, > created a topic with 21 partitions and 3 replicas. > > 2 producers are started with

Re: Auto Purging Consumer Group Configuration [Especially Kafka Console Group]

There is a JIRA open for exactly this issue with patch available: KAFKA-559 On Fri, Oct 10, 2014 at 9:33 AM, Neha Narkhede wrote: > Bhavesh, > > All consumer groups with at least one alive consumer will have a > registration node for that consume

Re: Create topic programmatically

A side note is that you may want to use waitUntil to check the topic is created after using AdminUtils.createTopic since it is async. On Mon, Oct 13, 2014 at 10:07 AM, Jonathan Weeks wrote: > Sure — take a look at the kafka unit tests as well as admin.AdminUtils , > e.g.: > > import kafka.ad

Re: Mirror Maker and Client ID

Hi Chad, Currently MM can only control which topics to MM from the source cluster, and hence your two data centers share the same topics it currently does not support that. The usual practice for your purpose is to set up 4 Kafka clusters, 2 on each data center (A and B), let's call them "local"

Re: Live Upgrade

Hi Balaji, You could do a rolling bounce of the brokers to do the in-place upgrade if your partitions have at least two replicas. After that you may probably need to rebalance the leaders if they are not balanced any more. Guozhang On Wed, Oct 15, 2014 at 10:53 AM, Seshadri, Balaji wrote: > Hi

Re: Live Upgrade

Yes. On Fri, Oct 17, 2014 at 8:52 AM, Roger Hoover wrote: > At least two including the leader? > > On Fri, Oct 17, 2014 at 8:12 AM, Guozhang Wang wrote: > > > Hi Balaji, > > > > You could do a rolling bounce of the brokers to do the in-place upgrade > if >

Re: frequent periods of ~1500 replicas not in sync

Neil, what you are seeing could probably be KAFKA-1407 . On Tue, Oct 21, 2014 at 12:03 PM, Gwen Shapira wrote: > Consumers always read from the leader replica, which is always in sync > by definition. So you are good there. > The concern would be

Re: Performance issues

This is a consumer config: fetch.wait.max.ms On Tue, Oct 21, 2014 at 11:39 AM, Mohit Anchlia wrote: > Is this a parameter I need to set it in kafka server or on the client side? > Also, can you help point out which one exactly is consumer max wait time > from this list? > > https://kafka.apache

Re: How many partition can one single machine handle in Kafka?

Xiaobin, This FAQ may give you some hints: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIchoosethenumberofpartitionsforatopic ? On Tue, Oct 21, 2014 at 12:15 AM, Xiaobin She wrote: > hello, everyone > > I'm new to kafka, I'm wondering what's the max num of partition can one >

Re: Strange behavior during un-clean leader election

Bryan, Did you take down some brokers in your cluster while hitting KAFKA-1028? If yes, you may be hitting KAFKA-1647 also. Guozhang On Mon, Oct 20, 2014 at 1:18 PM, Bryan Baugher wrote: > Hi everyone, > > We run a 3 Kafka cluster using 0.8.1.1 with all topics having a replication > factor of

Re: Explicit topic creation and topic metadata availability

Hello Stevo, Your understanding about the configs are correct, and it is indeed wired that the producer gets the exception after topic is created. Could you use the kafka-topics command to check if the leaders exist? kafka-topics.sh --zookeeper XXX --topic [topic-name] describe Guozhang On Wed,

Re: Erratic behavior when quickly re-balancing

Hello Eric, 1) The rebalance failures is mainly on ZK session timeout, you could try to increase your zk session timeout value and see if that helps. 2) The new consumer in 0.9 re-write will resolve this problem by getting rid of the ZK dependency and use a centralized coordinator for rebalance l

Re: Kafka 0.9

Hi Rajiv, We are currently working on checking in KAFKA-1583, which is a step 0 for the new consumer implementation. Once that is done, we will be in full spin for the coding. That said, it is fortunately not gonna happen by end of this year. The hope is to have a unit testable consumer by the end

Re: Kafka sending messages with zero copy

Rajiv, Could you let me know your use case? Are you sending a very large file and hence would prefer streaming manner instead of messages? Guozhang On Thu, Oct 23, 2014 at 4:03 PM, Rajiv Kurian wrote: > I have a flyweight style protocol that I use for my messages. Thus they > require no serial

Re: Kafka sending messages with zero copy

Rajiv, The new producer does maintain a buffer per partition, but you need to consider synchronizing the access to the buffer since it can take data from multiple caller threads. I think Jay's suggestion 1) does the same thing for your purpose if you already have the data buffer storing your data:

Re: Kafka sending messages with zero copy

I think 0.8.2 already used the new producer as the standard client. Guozhang On Fri, Oct 24, 2014 at 8:51 AM, Rajiv Kurian wrote: > Thanks I'll take a look at both. Just to be sure we are talking about > client version 0.82 right? > > > > On Fri, Oct 24, 2014 at 8:39 A

Re: Cannot connect to Kafka from outside of EC2

Could you see if this is the case: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers ? On Fri, Oct 24, 2014 at 11:35 AM, Sameer Yami wrote: > Hi, > > We have a Kafka / Zookeeper test setup on a single EC2 machine. The > host.name is the

Re: Explicit topic creation and topic metadata availability

>>> 21:12 $ bin/kafka-topics.sh --zookeeper 127.0.0.1:50194 --topic > >>> 059915e6-56ef-4b8e-8e95-9f676313a01c --describe > >>> Error while executing topic command null > >>> java.util.NoSuchElementException > >>> at scala.collection.IterableLike$cla

Re: Cannot connect to Kafka from outside of EC2

Test due to > kafka.common.LeaderNotAvailableException}] for topic [Test]: class > kafka.common.LeaderNotAvailableException > 2014-10-24 14:31:47,258 main ERROR > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by > topic, partition due to: Failed to fetch topic metadata f

Re: Kafka lost data

Hello Chen, You can look into brokers for "message size too large" exceptions if you cannot access the producer logs (both of them should have this in their log files). Also which ack mode are your producer using? Guozhang On Mon, Oct 27, 2014 at 10:31 AM, Mark Roberts wrote: > I think it will

Re: Explicit topic creation and topic metadata availability

vo Slavic. > > On Mon, Oct 27, 2014 at 12:43 AM, Guozhang Wang > wrote: > > > Hi Stevo, > > > > Sorry for the late reply. > > > > Yes, you have to use ZKStringSerializer when initializing zkClient. > However > > this is not related to key.seriali

Re: Cannot connect to Kafka from outside of EC2

: 0Isr: 0 > > > > The topics are being created at run-time. > > > > thanks > > > > > > On Sun, Oct 26, 2014 at 4:49 PM, Guozhang Wang > wrote: > > > >> Could you use the topic tools "--describe" to check if t

Re: Cannot connect to Kafka from outside of EC2

Isr: 0 > > > Topic: TestPartition: 1Leader: 0 Replicas: 0 > Isr: 0 > > > > > > thanks > > > Sameer > > > > > > > > > > > > > > > On Tue, Oct 28, 2014 at 9:40 AM, Guozhang Wang > > wrote: > >

Re: Cannot connect to Kafka from outside of EC2

thread-1 ERROR > kafka.producer.async.DefaultEventHandler-97: Failed to collate messages by > topic, partition due to: Failed to fetch topic metadata for topic: Test > 2014-10-28 21:46:25,167 pool-13-thread-1 INFO > kafka.producer.async.DefaultEventHandler-68: Back off for 1000 ms b

Re: Cannot connect to Kafka from outside of EC2

connecting to broker. > > (kafka.controller.RequestSendThread) > > java.nio.channels.ClosedChannelException > > at kafka.network.BlockingChannel.send(BlockingChannel.scala:89) > > at > > > kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelM

Re: Can both uncompressed data and compressed data reside in the same topic (partition)

Hi Zuoning, Yes, the same topic's partitions can hold both compressed and uncompressed data, and the consumer is able to read them in mixed mode. Guozhang On Fri, Oct 31, 2014 at 8:40 AM, Zuoning Yin wrote: > Hey Guys, > If at the beginning, I configure the producer to not use compressio

Re: Cannot connect to Kafka from outside of EC2

directory are > zookeeper-gc.log,kafkaServer-gc.log, server.log and state-change.log > > Thanks > > On Wed, Oct 29, 2014 at 4:13 PM, Guozhang Wang wrote: > > > Sorry for getting late on this Sameer. > > > > The logs you showed above are from controller log

Re: Kafka Cluster disaster decovery

Yingkai, Kafka uses persistent storage so the data written to it will not be lost, you just need to restart the cluster. But during the down time it will become un-available. Guozhang On Fri, Oct 31, 2014 at 2:06 PM, Yingkai Hu wrote: > Hi All, > > I’m new to Kafka, please direct me to the r

Re: Indefinite growth of FetchRequestPurgatory

Hi Andras, Could you try 0.8.2-beta and see if this issue comes out again? We fixed a couple of the purgatory issues (e.g. KAFKA-1616 ) in 0.8.2, but I do not remember any of them will cause OOM. Guozhang On Mon, Nov 3, 2014 at 5:42 AM, András Se

Re: question about console producer behavior

Hi, Producers will only refresh their metadata periodically if there is no exceptions caught sending the data, you can config this period in " topic.metadata.refresh.interval.ms" (default is 600 seconds). Guozhang On Mon, Nov 3, 2014 at 6:51 AM, raymond wrote: > Hi > > It appears to me that, w

Re: [ANNOUNCEMENT] Apache Kafka 0.8.2-beta Released

Koert, these two classes belong to the 0.9 consumer api, which are not dev ready yet. We only checked in the apis so people can review | comment on. Guozhang On Nov 1, 2014 8:26 AM, "Koert Kuipers" wrote: > joe, > looking at those 0.8.2 beta javadoc I also see a Consumer api and > KafkaConsumer

Re: Producer timeout setting not respected

Hello Solon, request.timeout.ms only controls the produce request timeout value, when the producer's first produce request gets timed out, it will try to re-fresh its metadata by sending metadata request. But when this non-produce request hits the broker whose connectivity has been disabled (i.e.

Re: Kafka 0.8.1.1 replication issues

This seems to be related to https://issues.apache.org/jira/browse/KAFKA-1749 . Guozhang On Tue, Nov 4, 2014 at 10:30 AM, Christofer Hedbrandh < christo...@knewton.com> wrote: > Hi Kafka users! > > I was just migrating a cluster of 3 brokers from one set of EC2 instances > to another, but ran int

Re: Producer timeout setting not respected

Actually I think this issue has just been resolved: https://issues.apache.org/jira/browse/KAFKA-1733 Guozhang On Tue, Nov 4, 2014 at 11:22 AM, Guozhang Wang wrote: > Hello Solon, > > request.timeout.ms only controls the produce request timeout value, when > the producer

Re: OffsetOutOfRange Error

Hi Jim, OffsetOutOfRange means that the partition's log offset range is [a, b] and the requested offset is either < a or > b. It could be caused by log truncation based on the retention policy while consumer fetching at the same time. Guozhang On Tue, Nov 4, 2014 at 4:21 PM, Jimmy John wrote:

Re: consumer ack for high-level consumer?

Hello, You can turn of auto.commit.offset and manually call connector.commitOffset() manually after you have processed the data. One thing to remember is that the commit frequency is related to ZK (in the future, Kafka) writes and hence you may not want to commit after processed every single messa

Re: How costly is Re balancing of partitions for a topic

Hello Dinesh, 1. A rebalance is triggered when the consumers is notified or the group member change / topic-partition change through ZK. 2. The cost of a rebalance is positively related to the #. consumers in the group and the #. of topics this group is consuming. The latency of the rebalance can

Re: How costly is Re balancing of partitions for a topic

ng running consumers? > > > > 2. You mentioned that rebalance latency depends on # of consumers and # > > number of topics. In the case of 1 topic and hundred of consumers can say > > the latency is in the tens of seconds as you mentioned before? > > > > 3. You mentione

Re: Cannot connect to Kafka from outside of EC2

etwork.Processor) > [2014-11-04 20:30:38,403] INFO Closing socket connection to /172.31.25.198 > . > (kafka.network.Processor) > [2014-11-04 20:30:39,410] INFO Closing socket connection to /172.31.25.198 > . > (kafka.network.Processor) > [2014-11-04 20:30:39,414] INFO Closing soc

Re: Consumer lag keep increasing

Chen, Your configs seems fine. Could you use ConsumerOffsetChecker tool to see if the offset is advancing at all (i.e. messages are comsumed), and if yes get some thread dumps and check if your consumer is blocked on some locks? Guozhang On Wed, Nov 5, 2014 at 2:01 PM, Chen Wang wrote: > Hey

Re: OffsetOutOfRange Error

Jimmy, I am not very familiar with the python java client, you may directly ask its author: https://cwiki.apache.org/confluence/display/KAFKA/Clients On Thu, Nov 6, 2014 at 7:57 AM, Jimmy John wrote: > I dug deeper and saw this during normal operation: > > In the kafka broker log: > > [2014-11

Re: consumer ack for high-level consumer?

mitOffset is called, local > offsets are sync to ZK. Is it correct? > > regards, > Chia-Chun > > 2014-11-06 0:24 GMT+08:00 Guozhang Wang : > > > Hello, > > > > You can turn of auto.commit.offset and manually call > > connector.commitOffset() manuall

Re: corrupt recovery checkpoint file issue....

Jason, Yes I agree with you. We should handle this more gracefully as the checkpoint file dump is not guaranteed atomic. Could you file a JIRA? Guozhang On Thu, Nov 6, 2014 at 6:31 AM, Jason Rosenberg wrote: > Hi, > > We recently had a kafka node go down suddenly. When it came back up, it > ap

Re: Consumer and Producer configs

Hello Eduardo, If you are using console producer / consumer, you can set the configs in command line starting them; if you are wiring the clients directly, then you can set them in a Properties and pass them to the constructor. Guozhang On Thu, Nov 6, 2014 at 7:10 AM, Eduardo Costa Alfaia wrote

Re: consumer ack for high-level consumer?

ed, local offsets >sync to ZK >3. Client fails when processing this message >4. Client restarts, but this message is marked as consumed in ZK > > Thanks, > Chia-Chun > > 2014-11-07 1:45 GMT+08:00 Guozhang Wang : > > > That is correct. > > >

Re: Cannot connect to Kafka from outside of EC2

Sameer, The server logs do not contain any non-INFO logs, which is a bit wired. Did you deploy the current trunk of Kafka? Also could you enable DEBUG level logging on Kafka brokers? Guozhang On Wed, Nov 5, 2014 at 3:50 PM, Sameer Yami wrote: > The server.log was taken separately. > We ran the

Re: corrupt recovery checkpoint file issue....

Jun, Checking the OffsetCheckpoint.write function, if "fileOutputStream.getFD.sync" throws exception it will just be caught and forgotten, and the swap will still happen, may be we need to catch the SyncFailedException and re-throw it as a FATAIL error to skip the swap. Guozhang On Thu, Nov 6,

Re: OffsetOutOfRange errors

Hi Jim, When messages gets cleaned based on data retention policy (by time or by size), the brokers will not inform ZK for the deletion event. The underlying assumption is that when consumers are fetching data at around the tail of the log (i.e. they are not much lagging, which is normal cases) th

Re: OffsetOutOfRange errors

bytes. So we will read in a bunch of > messages in a batch. And while iterating through those messages, we commit > the offset to ZK every 100. > > jim > > On Fri, Nov 7, 2014 at 10:13 AM, Guozhang Wang wrote: > > > Hi Jim, > > > > When messages gets clean

Re: Interrupting controlled shutdown breaks Kafka cluster

Solon, Which version of Kafka are you running and are you enabling auto leader rebalance at the same time? Guozhang On Fri, Nov 7, 2014 at 8:41 AM, Solon Gordon wrote: > Hi all, > > My team has observed that if a broker process is killed in the middle of > the controlled shutdown procedure, th

Re: Interrupting controlled shutdown breaks Kafka cluster

ce.enable=true. > > On Fri, Nov 7, 2014 at 2:35 PM, Guozhang Wang wrote: > > > Solon, > > > > Which version of Kafka are you running and are you enabling auto leader > > rebalance at the same time? > > > > Guozhang > > > > On Fri, Nov 7, 2014

Re: Error in fetch Name. How to recover broken node?

Hi Marco, The fetch error comes from "UnresolvedAddressException", could you try to check if you have a network partition issue during that time? As for the "Too many file handlers", I think this is due to not properly handling such exceptions that it does not close the socket in time, which vers

Re: One question about "New Producer Configs"

Just some additions to Chia-Chun's response: each topic can have multiple partitions and each partition can be replicated as multiple replicas on different machines, acks = n means that the data sent to a particular partition has been replicated to at least n replicas. Guozhang On Sun, Nov 9, 201

Re: Error in fetch Name. How to recover broken node?

sing kafka 0.8.1.1. > > About network partition, it is an option. > now i'm just wondering if deleting the data folder on the second node will > at least have it come up again. > > i think another guy tried a kafka-reassign-partitions just before it all > blew up. > > >

Re: corrupt recovery checkpoint file issue....

xceptions. There is only a > finally clause to close the writer. So, it there is any exception during > write or sync, the exception will be propagated back to the caller and > swapping will be skipped. > > Thanks, > > Jun > > On Fri, Nov 7, 2014 at 9:47 AM, Guozhan

Re: Cannot connect to Kafka from outside of EC2

gt; > > >> The version is kafka_2.10-0.8.1.1. It is not the latest trunk. > >> Will try enabling debug version. > >> > >> thanks > >> > >> > >> On Thu, Nov 6, 2014 at 9:37 PM, Guozhang Wang > wrote: > >> > >>>

Re: first message is lost

Yonghui, If consumer is not set with --from-beginning, then this scenario is expected: KAFKA-1006 We are still figuring what is the best way to resolve this issue. Guozhang On Wed, Nov 12, 2014 at 12:35 AM, Yonghui Zhao wrote: > Hi, > > For a

Re: Programmatic Kafka version detection/extraction?

Sounds great, +1 on this. On Tue, Nov 11, 2014 at 1:36 PM, Gwen Shapira wrote: > So it looks like we can use Gradle to add properties to manifest file and > then use getResourceAsStream to read the file and parse it. > > The Gradle part would be something like: > jar.manifest { > att

Re: Order of consumed messages in the same partition.

Hi Filippo, By saying "offset" do you mean the offset field inside the message or you keep track of the ordering from the producer end and found it not consistent as observed from the consumer end? Guozhang On Wed, Nov 12, 2014 at 8:39 AM, Filippo De Luca wrote: > Hi all, > I would like to kno

Re: 0.8.2 producer with 0.8.1.1 cluster?

Shlomi, It should be compatible, did you see any issues using it against a 0.8.1.1 cluster? Guozhang On Wed, Nov 12, 2014 at 5:43 AM, Shlomi Hazan wrote: > Hi, > Is the new producer 0.8.2 supposed to work with 0.8.1.1 cluster? > Shlomi > -- -- Guozhang

Re: Broker keeps rebalancing

Hey Chen, As Neha suggested, typical reason of too many rebalances is that your consumers kept being timed out from ZK, and you can verify this by checking in your consumer logs for sth. like "session timeout" entries (these are not ERROR entries). Guozhang Guozhang On Wed, Nov 12, 2014 at 5:31

Re: Broker keeps rebalancing

myid:1] - INFO [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for > client /10.93.80.121:38437 which had sessionid 0x149a4cc1b580e7e > > We are using -Xmx2048m for consumer, and I didn't see any GC related > exceptions > > Chen >

Re: How to keep-alive connection from producer to broker?

Neeraja, Producer does use keep-alive connections to the brokers, and a recent change is introduced in broker which will actively close connections if it has not got any requests from the producer for some time. The default period is 10 min, you can set it to INT_MAX if you do not want this featur

Re: Broker keeps rebalancing

om:2888:3888 > > server.2=.com:2888:3888 > > server.3=.com:2888:3888 > > > On Thu, Nov 13, 2014 at 10:27 AM, Guozhang Wang > wrote: > > > Chen, > > > > From ZK logs it sounds like ZK kept timed out consumers which triggers > > rebalance. &g

Re: Broker keeps rebalancing

eck your consumer GC log to see if it is paused from time to time. Guozhang On Thu, Nov 13, 2014 at 11:01 AM, Chen Wang wrote: > kafka.config.zookeeper.session.timeout.ms6 > kafka.config.rebalance.backoff.ms6000kafka.config.rebalance.max.retries6 > > On Thu, Nov 13, 2014 at 10:56 AM

Re: How to keep-alive connection from producer to broker?

, Neeraja (HP Software) < neeraj...@hp.com> wrote: > @Guozhang, > > Thanks for your reply. > Is is available as part of broker configuration? If yes, how to configure. > > --Neeraja > > > -----Original Message- > From: Guozhang Wang [mailto:wangg...@gmail.com]

Re: will adding partitions to existing topic change leadership to existing partitions?

Hi Shlomi, As long as your json file indicating the partition addition operation does not touch the existing partitions you should be fine. Guozhang On Mon, Nov 17, 2014 at 1:08 AM, Shlomi Hazan wrote: > Hi > I want to add partitions to a running topic, > and since I use the python producer I

Re: Kafka broker cannot start after running out of disk space

This is interesting as I have not seen it before. Searched a bit on the web and this seems promising? http://stackoverflow.com/questions/2949371/java-map-nio-nfs-issue-causing-a-vm-fault-a-fault-occurred-in-a-recent-uns Guozhang On Fri, Nov 14, 2014 at 5:38 AM, Yury Ruchin wrote: > Hello, > >

Re: partition auto-rebalance

Hello Wes, The document here is a bit misleading indeed: http://kafka.apache.org/documentation.html#brokerconfigs In Kafka a partition has a replica list {A,B,C..} and the first replica would be the leader of the partition. When it is not the case, for example since A is down B becomes the leade

Re: Topic Creation

We have a ticket open for making admin tools like topic creation to be sync: KAFKA-1125 On Tue, Nov 18, 2014 at 9:25 PM, Jun Rao wrote: > Yes, that's the best option for now. The creation is async. > > Thanks, > > Jun > > On Tue, Nov 18, 2014 at

Re: will adding partitions to existing topic change leadership to existing partitions?

ics.sh --zookeeper localhost:2181 --alter --topic test_topic > --partitions 8 > Will then existing partitions be subject to leadership change? > 10x > Shlomi > > On Mon, Nov 17, 2014 at 7:04 PM, Guozhang Wang wrote: > > > Hi Shlomi, > > > > As long as your j

Re: Drawbacks to setting log.roll.hours=1 ?

Hello, One thing you need to check is that the file handler limit on your machine is set high enough for the number of files on disk plus sockets. Other than that, as long as you are consuming at the tail of the log most of the time it should be fine. Guozhang On Wed, Nov 19, 2014 at 9:23 AM, Th

Re: BytesInPerSec greater than BytesOutPerSec

Are you having consumers fetching from these topics at the same time? BytesInPerSec only counts the bytes appended to the log from the produce request, and BytesOutPerSec counts the bytes fetch from fetch requests; hence both replica fetcher and normal consumer fetcher's requests count in BytesOut

Re: How to recover from ConsumerRebalanceFailedException ?

Bhavesh, In 0.9 consumer would not talk to ZK and will be single threaded, which will be easier to provide monitoring mechanisms. Guozhang On Thu, Nov 20, 2014 at 8:15 PM, Jun Rao wrote: > Can you just monitor the consumer byte/message/fetch rate? > > Thanks, > > Jun > > On Thu, Nov 20, 2014 a

Re: BytesInPerSec greater than BytesOutPerSec

ec on the broker given that > we have active consumer and replication. > > > On Fri, Nov 21, 2014 at 4:32 PM, Guozhang Wang wrote: > > > Are you having consumers fetching from these topics at the same time? > > > > BytesInPerSec only counts the bytes appended to

Re: How many messages does each broker have?

Palur, If the 8 partitions are hosted on each one of the nodes, assuming replication factor 1 then each node will get roughly 10 / 8 messages due to the random partitioner. If you want to know exactly how many messages is on each broker then you can use a simple consumer which allows you to sp

Re: MetadataResponse error code handling

Hi Evan, The error code is per-partition, and for new producer only LeaderNotAvailable error will cause it to ignore the metadata for that partition. If you are using the producer clients directly you would not need to worry about these error handling details; if you are fetching the metadata usin

Re: Can Mirroring Preserve Every Topic's Partition?

Hello Alex, This can be done by doing some tweaks in the MM code (with the 0.8.2 new producer). 1. Set-up your MM to have the total # of producers equal to the #. of partitions in source / target cluster. 2. When the consumer of the MM gets a message, put the message to the producer's queue base

Re: MetadataResponse error code handling

both of these ignored? Or > is it that LeaderNotAvailable causes the metadata to be ignored, and *all* > other partition-specific errors (including ReplicaNotAvailable) still carry > legitimate data? > > Thanks, > Evan > > [1] https://github.com/Shopify/sarama > > On

Re: Pagecache cause OffsetOutOfRangeException

Yuanjia, I am not sure that pagecache can be the cause of this, could you attach your full stack trace and use the GetOffset tool Manikumar mentioned to make sure the offset does exist in the broker? Guozhang On Tue, Dec 2, 2014 at 7:50 AM, Manikumar Reddy wrote: > You can check the latest/ear

  1   2   3   4   5   6   7   8   9   10   >