I need some help with the production server architecture

Hi, Sometime back i was informed on the group that in production we should never run kafka on same physical machine. So based on that I have a question on how to divide the server nodes we have to run zookeper and kafka brokers. I have a following setup Data center 1 Lan 1 (3 VMs) 192.168.xx.yy1 1

Question about state store

Hello all, I'm not sure how to do the following use-case, if anyone can help :) - I have an admin UI where the admin can choose wich item are allowed on the application - When the admin choose an item, it pushes an object to a topic kafka : test.admin.item with the following object {"id":"12

Two possible issues with 2.11 0.10.1.0

Ran into these two internal exceptions today. Likely they are known or my fault but I'm reporting them anyway: Exception in thread "StreamThread-1" java.lang.NullPointerException at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:304) at org.apache.kafka.stre

Re: while publishing message need to add multiple keys in a single message

Kafka doesn't have the concept of message headers like some other messaging systems. You will have to create a payload that contains these headers and whatever bytes you are sending. Dave > On Nov 28, 2016, at 16:47, Prasad Dls wrote: > > Hi, > > While publishing each message (single message

HOW TO GET KAFKA CURRENT TOPIC GROUP OFFSET

HI! WAIT YOUR ANSWER

Re: Kafka Streaming

I just cloned 3.1x and tried to run a test. I am still seeing rocksdb error: Exception in thread "StreamThread-1" java.lang.UnsatisfiedLinkError: C:\Users\manchlia\AppData\Local\Temp\librocksdbjni108789031344273.dll: Can't find dependent libraries On Mon, Oct 24, 2016 at 11:26 AM, Matthias J

Re: Kafka Connect consumer not using the group.id provided in connect-distributed.properties

Hi, I am using Kafka Connect Sink Connector to consume messages from a Kafka topic in a secure Kafka cluster. I have provided the group.id in connect-distributed.properties. I am using security.protocol as SASL_PLAINTEXT. Here is definition of group id in connect-distributed.properties - bootstra

Kafka Connect consumer not using the group.id provided in connect-distributed.properties

Hi, I am using Kafka Connect Sink Connector to consume messages from a Kafka topic in a secure Kafka cluster. I have provided the group.id in connect-distributed.properties. I am using security.protocol as SASL_PLAINTEXT. Here is definition of group id in connect-distributed.properties - [sa9726@

Re: Initializing StateStores takes *really* long for large datasets

I'll write an update on where I am now. I've got about 40 'primary' topics, some small, some up to about 10M messages, and about 30 internal topics, divided over 6 stream instances, all running in a single app, talking to a 3 node Kafka cluster. I use a single thread per stream instance, as my pr

Kafka Clients Survey

Hey Kafka Community, I'm trying to take a pulse on the current state of the Kafka clients ecosystem. Which languages are most popular in our community? What does the community value in clients? You can help me out by filling in the survey: https://goo.gl/forms/cZg1CJyf1PuqivTg2 I will lock the s

while publishing message need to add multiple keys in a single message

Hi, While publishing each message (single message) to Kafka, I need to add below headers/key ID: 123456 TYPE: xyz EVENTE: A OPERATION: Insert createTimeStampt: 2016-11-24T19:41:23.354Z updatedTimeStamp: 2016-11-30T19:41:23.354Z Please help me on this, how can i add all these into single messa

Re: kafka 0.10.1 ProcessorTopologyTestDriver issue with .map and .groupByKey.count

Hamid, would you mind creating a Jira? Thanks. -Matthias On 11/28/16 9:36 AM, Guozhang Wang wrote: > Damian, Hamid: > > I looked at the source code and suspect that it is because of the > auto-repartitioning which causes the topology to not directly forward the > record to the child processors

Re: Data (re)processing with Kafka (new wiki page)

Thanks for your input! For (2), there is currently KafkaStream.toString(). You could theoretically parse the String to extract all required information -- of course, this is not a nice solution. -Matthias On 11/25/16 2:11 AM, saiprasad mishra wrote: > This page is really helpful.Thanks for putti

Re: Kafka 0.10.1.0 consumer group rebalance broken?

There has been plenty of changes in the GroupCoordinator and co between these two releases: https://github.com/apache/kafka/commit/40b1dd3f495a59abef8a0cba5450526994c92c04#diff-96e4cf31cd54def6b2fb3f7a118c1db3 It might be related to this: https://github.com/apache/kafka/commit/4b003d8bcfffded55a00

Kafka 0.10.1.0 consumer group rebalance broken?

Hi, It seems that consumer group rebalance is broken in Kafka 0.10.1.0 ? When running a small test-project : - consumers running in own JVM (with different 'client.id') - producer running in own JVM - kafka broker : the embedded kafka : KafkaServerStartable It looks like the consumers loose their

Re: Initializing StateStores takes *really* long for large datasets

Hello Frank, How many instances do you have in your apps and how many threads did you use per thread? Note that besides the topology complexity (i.e. number of state stores, number of internal topics etc) the (re-)initialization process is depending on the underlying consumer's membership protocol

Re: KafkaStream: puncutuate() never called even when data is received by process()

Yes we are considering to differentiate "process" and "punctuate" function to be "data-driven" and "time-driven" computations. That is, triggering of punctuate should NOT be depending on the arrival of messages, or the message's associated timestamps. As for now I think periodically inserting the

Problem consuming using 0.10.1.0

(caveat - was having this issue with 0.10.0 but thought an update might help) App has a simple layout: consume -> aggregate -> output to 2nd topic Starting the app I see pages of this sort of message: DEBUG org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0xb58abf2daaf0002 aft

RE: Kafka consumers are not equally distributed

Thank you, Guozhang. I see a lot of this exception: org.apache.kafka.clients.consumer.internals.ConsumerCoordinator::Offset commit failed. : TimeoutException: The request timed out. We are committing the offset manually by Asynch mode and session.timeout.ms is 5 mins and poll time 10 secs and

Re: Kafka windowed table not aggregating correctly

Sachin, This is indeed a bit wired, and we'd like to try to re-produce your issue locally. Do you have a sample input data for us to try out? Guozhang On Fri, Nov 25, 2016 at 10:12 PM, Sachin Mittal wrote: > Hi, > I fixed that sorted set issue but I am facing a weird problem which I am > not a

Re: Interactive Queries

Thanks All. On Mon, Nov 28, 2016 at 3:09 AM, Michael Noll wrote: > There are also some examples/demo applications at > https://github.com/confluentinc/examples that demonstrate the use of > interactive queries: > > - > https://github.com/confluentinc/examples/blob/3. > 1.x/kafka-streams/src/main

Re: kafka 0.10.1 ProcessorTopologyTestDriver issue with .map and .groupByKey.count

Damian, Hamid: I looked at the source code and suspect that it is because of the auto-repartitioning which causes the topology to not directly forward the record to the child processors but send to an intermediate topic. In our tests we only do "groupByKey" without map, and hence auto-repartitioni

Re: A strange controller log in Kafka 0.9.0.1

thanks to Jason Gustafson, hope more contributor can take part in this discussion. https://issues.apache.org/jira/browse/KAFKA-4447 > 在 2016年11月27日，下午9:20，Json Tu 写道： > > AnyBody？This is very disconcerting! If convenient, Can somebody help to

Re: --group flag for console consumer

Hi Andrew, The PR can be found here: https://github.com/apache/kafka/pull/2150 Thanks. --Vahid From: Andrew Pennebaker To: users@kafka.apache.org Date: 11/27/2016 01:08 PM Subject:Re: --group flag for console consumer Vahid, I'd like to see your PR if you please! Confluent

RE: Oversized Message 40k

> If you have performance numbers you can share for the large messages, I think > we'll all appreciate :) Here is a sample test run: 3 brokers: 256G memory, 32 cores 3 zookeeper on smaller VMs 15 topics, replication: 3, partitions: 3 I had two JMeter instances sending "blobs" of random X

Re: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

I can’t see the code, I think it’s not added to the email? Thanks Eno From: Hamza HACHANI Reply-To: Date: Monday, 28 November 2016 at 13:25 To: "users@kafka.apache.org" Subject: RE: Abnormal working in the method punctuate and error linked to seesion.timeout.ms Hi Eno, Here is the co

Re: Messages intermittently get lost

Thanks Martin, I will look at those links. But you seem to be 100% confident that the problem is with ZooKeeper...can I ask why? What is it about my problem description that makes you think this is an issue with ZooKeeper? From: Martin Gainty Sent: Friday, Nov

RE: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

the print is in line 40 of the class Base... De : Hamza HACHANI Envoyé : lundi 28 novembre 2016 01:25:08 À : users@kafka.apache.org Objet : RE: Abnormal working in the method punctuate and error linked to seesion.timeout.ms Hi Eno, Here is the code for the app

RE: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

Hi Eno, Here is the code for the application ExclusiveStatsConnectionDevice which is composed of 4 nodes. For example when i put print("") you would sess the problem of the infinite loop. I preferred to send the whole code sto make it easier to you even though you don't need all of it _

Re: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

Hi Hamza, Would you be willing to share some of your code so we can have a look? Thanks Eno > On 28 Nov 2016, at 12:58, Hamza HACHANI wrote: > > Hi Eno. > > The problem is that there is no infinite while loop that i write. > > So I can't understand why the application is doing so. > > > Ham

RE: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

Hi Eno. The problem is that there is no infinite while loop that i write. So I can't understand why the application is doing so. Hamza De : Eno Thereska Envoyé : dimanche 27 novembre 2016 23:21:24 À : users@kafka.apache.org Objet : Re: Abnormal working in the

Re: Abnormal working in the method punctuate and error linked to seesion.timeout.ms

Hi Hamza, If you have an infinite while loop, that would mean the app would spend all the time in that loop and poll() would never be called. Eno > On 28 Nov 2016, at 10:49, Hamza HACHANI wrote: > > Hi, > > I've some troubles with the method puctuate.In fact when i would like to > print a

Abnormal working in the method punctuate and error linked to seesion.timeout.ms

Hi, I've some troubles with the method puctuate.In fact when i would like to print a string in the method punctuate. this string would be printed in an indefinitly way as if I printed (while (true){print(string)}. I can't understand what happened.Does any body has an explenation ?. Besides I

Re: 0.8 consumers compatibility with Kafka 0.10

Hi Vladi, Generally, newer brokers are compatible with older clients. As Vladimir said, the upgrade page (http://kafka.apache.org/documentation.html#upgrade) includes detailed information about changes that could have a compatibility impact. Ismael On Sun, Nov 27, 2016 at 8:09 PM, Vladi Feigin

Re: Interactive Queries

There are also some examples/demo applications at https://github.com/confluentinc/examples that demonstrate the use of interactive queries: - https://github.com/confluentinc/examples/blob/3.1.x/kafka-streams/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExampl