RE: Mirror multi-embedded consumer's configuration

Hi, Anybody could help to answer below question? Thanks! Best Regards Johnny From: ZHU Hua B Sent: 2016年10月19日 16:22 To: 'users@kafka.apache.org' Subject: Mirror multi-embedded consumer's configuration Hi, I launch Kafka mirror maker with multi-embedded consumer's configuration but

Re: Offset sporadically being reset

Additionally, on consumer I observe a strange behavior: it is being constantly rebalancing. There are no errors and each rebalance succeeds, but as soon as one is finished the next one is started. On Wed, Oct 19, 2016 at 4:36 PM, Timur Fayruzov wrote: > Hello, > > I

Error while executing consumer group command null and Exiting due to: empty.head error while displaying offsets

Hi , I am using Kafka 0.9 broker and 0.8 Consumer. Consumer was running fine since long time but today, I am getting the below exception. [ConsumerFetcherThread--5], Error for partition [rtdp.bizlogging.iditextlogdata,93] to broker 5:class kafka.common.NotLeaderForPartitionException If I

Offset sporadically being reset

Hello, I run Kafka 0.8.2.2 cluster with 3 nodes and recently started to observe strange behavior on select topics. The cluster runs in-house as well as most consumers. I have started some consumers in AWS and they _mostly_ work fine. Occasionally, I end up in a state where when I run

Re: Frequent UNKNOWN_MEMBER_ID errors in kafka consumer

This usually means that the consumer is being kicked out of the group due to heartbeat failure detection protocol. Try to increase your session.timeout.ms config so that the heartbeating is less sensitive, and see if this issue happens less frequently. Guozhang On Wed, Oct 19, 2016 at 12:59

Re: kafka streams metadata request fails on 0.10.0.1 broker/topic from 0.10.1.0 client

Thanks Michael Hopefully the upgrade story evolves as 0.10.1+ advances to maturity Just my 2 cents Decoupling the kafka streams from the core kafka changes will help so that the broker can be upgraded without notice and streaming apps can evolve to newer streaming features on their own pace

Re: Mantain only last message by key

You can try cleanup.policy=compact. But be careful with a large number of keys. – Best regards, Radek Gruchalski ra...@gruchalski.com On October 19, 2016 at 11:44:39 PM, Jesus Cabrera Reveles ( jesus.cabr...@encontrack.com) wrote: Hello, We are a company of IoT and we are trying to implement

Mantain only last message by key

Hello, We are a company of IoT and we are trying to implement kafka but we have some questions. We need a topic with a certain number of partitions, each message has its own key, relative to id device. And we need the topic hold only the last message, we don't need historic of messages

Re: Please help with AWS configuration

Sorry, had a typo in my gist. Here is the correct location: https://gist.github.com/anduill/710bb0619a80019016ac85bb5c060440 On 10/19/16, 4:33 PM, "David Garcia" wrote: Hello everyone. I’m having a hell of a time figuring out a Kafka performance issue in AWS. Any

Please help with AWS configuration

Hello everyone. I’m having a hell of a time figuring out a Kafka performance issue in AWS. Any help is greatly appreciated! Here is our AWS configuration: - Zookeeper Cluster (3.4.6): 3-nodes on m4.xlarges (default configuration) EBS volumes (sd1) - Kafka Cluster (0.10.0):

Re: How to block tests of Kafka Streams until messages processed?

Yeah, I did think to use that method, but as you said, it writes to a dummy output topic, which means I'd have to put in magic code just for the tests to pass (the actual code writes to cassandra and not to a dummy topic). On Thu, Oct 20, 2016 at 2:00 AM, Tauzell, Dave

RE: How to block tests of Kafka Streams until messages processed?

For similar queue related tests we put the check in a loop. Check every second until either the result is found or a timeout happens. -Dave -Original Message- From: Ali Akhtar [mailto:ali.rac...@gmail.com] Sent: Wednesday, October 19, 2016 3:38 PM To: users@kafka.apache.org Subject:

Re: How to block tests of Kafka Streams until messages processed?

Wanted to add that there is nothing too special about these utility functions, they are built using a normal consumer. Eno > On 19 Oct 2016, at 21:59, Eno Thereska wrote: > > Hi Ali, > > Any chance you could recycle some of the code we have in >

Re: How to block tests of Kafka Streams until messages processed?

Hi Ali, Any chance you could recycle some of the code we have in streams/src/test/java/.../streams/integration/utils? (I know we don't have it easily accessible in Maven, for now perhaps you could copy to your directory?) For example there is a method there

Re: Embedded Kafka Cluster - Maven artifact?

Please change that. On Thu, Oct 20, 2016 at 1:53 AM, Eno Thereska wrote: > I'm afraid we haven't released this as a maven artefact yet :( > > Eno > > > On 18 Oct 2016, at 13:22, Ali Akhtar wrote: > > > > Is there a maven artifact that can be used

Re: Embedded Kafka Cluster - Maven artifact?

I'm afraid we haven't released this as a maven artefact yet :( Eno > On 18 Oct 2016, at 13:22, Ali Akhtar wrote: > > Is there a maven artifact that can be used to create instances > of EmbeddedSingleNodeKafkaCluster for unit / integration tests?

How to block tests of Kafka Streams until messages processed?

I'm using Kafka Streams, and I'm attempting to write integration tests for a stream processor. The processor listens to a topic, processes incoming messages, and writes some data to Cassandra tables. I'm attempting to write a test which produces some test data, and then checks whether or not the

ApacheCon is now less than a month away!

Dear Apache Enthusiast, ApacheCon Sevilla is now less than a month out, and we need your help getting the word out. Please tell your colleagues, your friends, and members of related technical communities, about this event. Rates go up November 3rd, so register today! ApacheCon, and Apache Big

Re: kafka streams metadata request fails on 0.10.0.1 broker/topic from 0.10.1.0 client

Apps built with Kafka Streams 0.10.1 only work against Kafka clusters running 0.10.1+. This explains your error message above. Unfortunately, Kafka's current upgrade story means you need to upgrade your cluster in this situation. Moving forward, we're planning to improve the

Re: Kafka Streams Aggregate By Date

I could successfully get the total (not average). As far as I see, there is no need to create a topic manually before I run the app. Topic is created if there is data and topic name not exists. Here is my code: KStreamBuilder builder = new KStreamBuilder(); KStream longs

Re: [VOTE] 0.10.1.0 RC3

+1 from myself too. The vote passes with 9 +1 votes and no 0 or -1 votes. +1 votes PMC Members: * Gwen Shapira * Jun Rao * Neha Narkhede Committers: * Ismael Juma * Jason Gustafson Community: * Eno Thereska * Manikumar Reddy * Dana Powers * Magnus Edenhill 0 votes * No votes -1 votes * No

Re: [VOTE] 0.10.1.0 RC3

+1 (non-binding) passes librdkafka test suites 2016-10-19 15:55 GMT+02:00 Ismael Juma : > +1 (non-binding). > > Verified source and Scala 2.11 binary artifacts, ran ./gradlew test with > JDK 7u80, quick start on source artifact and Scala 2.11 binary artifacts. > > Thanks for

Re: Need to add & remove consumers dynamically in diffrent group and consume consecutively.

Do you have only one partition in the topic? The way Kafka works is that all messages are first distributed into partitions in the topic and then the consumers are distributed among them and they read them sequentially.  If you have only one partition in the topic, all your messages will be in

Need to add & remove consumers dynamically in diffrent group and consume consecutively.

Hi guys, i am using apache kafka with phprd kafka, i want to know how can i use multiple Kafka consumers on same partition from different groups to consume message parallel, say if consumer are c1,c2,c3 consuming single partition 0, than if c1 is consuming from 0 offset than c2 should start from 1

A more reliable way of consumer leaving group when it is closed?

Hi. I’ve run into a problem with Kafka consumers not leaving their consumer group cleanly when the application restarts. Out of 9 topics that (with a consumer for each) it seems that every time at least 2 or 3 do not leave the group cleanly so when the application starts these consumers start

ZK WARN Cannot open channel to 3 at election address .... java.net.ConnectException: Connection refused

Hi, I am running ZK from version 0.10.0.1 on 3 machines, the hostnames haven't been registered on DNS so all defined in /etc/hosts, there are no firewall between them. I saw many posts online pointed to some bugs in the zookeepers and there are many patches to fix it. Has the zookeeper in

Getting issue with offset commit

Hi Initially we are using 8.2 Kafka client and server and things are working fines. Now Kafka server is upgraded to 10.0 and we getting an issue with offset commit. *Now Kafka setup looks like:* Kafka client : 8.2 version Kafka server: 10.2 version we are using manual offset commit and we are

Mirror multi-embedded consumer's configuration

Hi, I launch Kafka mirror maker with multi-embedded consumer's configuration but failed as below, what's the mean of "you asked for only one", is there an option control it? Thanks! # bin/kafka-mirror-maker.sh --consumer.config config/consumer-1.properties --consumer.config

Re: Kafka consumer cluster setup

Hi Ian, Thanks for the detailed explanation. Thanks and Regards A.SathishKumar > Hi > The client initially connects to the first Broker specified in > bootstrap.servers (if that’s > not available, it’ll connect to the next one on the list, and so on). When it > does so, it > is then given

Frequent UNKNOWN_MEMBER_ID errors in kafka consumer

Hi, I have a consumer which implements new consumer api (0.9.0.1). I see below errors quite frequently in the consumer application logs: ERROR [pool-4-thread-5] - o.a.k.c.c.i.ConsumerCoordinator - Error UNKNOWN_MEMBER_ID occurred while committing offsets for group audit.consumer.group Can you

Re: [kafka-clients] [VOTE] 0.10.1.0 RC3

+1 (binding) Verified quick start and artifacts. On Mon, Oct 17, 2016 at 10:39 PM Dana Powers wrote: > +1 -- passes kafka-python integration tests > > On Mon, Oct 17, 2016 at 1:28 PM, Jun Rao wrote: > > Thanks for preparing the release. Verified quick