Question regarding Kafka Streams Global State Store

Good morning, I have setup a Kafka Streams application with the following logic. The incoming messages are validated and transformed. The transformed messages are then published to a global state store via topic A as well as to an additional topic A for consumption by other applications

Re: Question regarding Kafka Streams Global State Store

Hi Georg, >From your description, I do not see why you need to use a global state instead of a local one. Are there any specific reasons for that? With a local state store you would have the previous record immediately available. Best, Bruno On Tue, May 19, 2020 at 10:23 AM Schmidt-Dumont Georg

AW: Question regarding Kafka Streams Global State Store

Hi Bruno, Thanks for your quick reply! I decided to use a global state store for two reasons. If the application crashes, the store is populated properly once the reason for the crash has been fixed and the app starts again, i.e. I feel that it gives me a certain resiliency. Second we will be

Cannot access to kafka by server domain and port.

Hi, I have been struggling for this connection problem for a whole week. I run Kafkaon my server machine which is on Cloud. And I cannot manage to connect to Kafka in anyway. It always print out `Connection to node -1 (myServerDomain.ltd/myServerIp:9092) could not be established. Broker may not

Re:Cannot access to kafka by server domain and port.

It seems that your OUTSIDE listener is wrong. "OUTSIDE://localhost:9092" means 127.0.0.1:9092. Try "OUTSIDE://0.0.0.0:9092" At 2020-05-19 17:57:41, "深大李展发" wrote: Hi, I have been struggling for this connection problem for a whole week. I run Kafka on my server machine which is on

Re: Question regarding Kafka Streams Global State Store

Hi Georg, local state stores in Kafka Streams are backed by a Kafka topic by default. So, if the instance crashes the local state store is restored from the local state directory. If the local state directory is empty or does not exist the local state store is restored from the Kafka topic. Local

??????Cannot access to kafka by server domain and port.

Thank you, Ixy, thanks for your reply, it's working now! I will only need to try to set it up in Docker. May I ask why this is different? I had also tried "OUTSIDE://:9092" before, is there any difference between these config? I am always confused by this config, I thought it only means Kafka

?????? Cannot access to kafka by server domain and port.

Hi, Moffatt, I had read this article many times, I am horrible and know few in Computer Network, it indeed explains how to configure Kafka to work in AWS and Docker, I will read again. Thank you. ---- ??:"Robin

AW: Question regarding Kafka Streams Global State Store

Hi Bruno, The data I am processing comes from machines. The key is the identifier for the machine which produced a specific message. Currently we only have a couple of these machines producing data. This number will increase quite a lot over the coming years. Is the limit on the number of

AW: Question regarding Kafka Streams Global State Store

Hi Bruno, I just had a discussion with a colleague of mine regarding this and I wanted to give you a quick contextual update. With regards to the global state, I realize that having this state consistent in a distributed system is very difficult. My expectation was that since it is a global

Re: Question regarding Kafka Streams Global State Store

Hi Georg, Great that you could answer your own question and I am glad that I could help. I was just writing you a similar answer. Yes, the global state store will eventually reflect your write but you do not know when. That is the main issue for your use case. A local state store will

KSQL to search for data in Kafka Topics

Hello, I am quite new to KSQL, so apologise for misunderstanding it's concept. I have a list of topics that I want to search data for. I am not using stream process, but plain topics which has data retained for 14 days. All i want to do is search for data in SQL-like way as long as it's within

KTable as a compacted topic, implications

This is the topology of a simple word count: Topologies: Sub-topology: 0 Source: KSTREAM-SOURCE-00 (topics: [word_count_input]) --> KSTREAM-FLATMAPVALUES-01 Processor: KSTREAM-FLATMAPVALUES-01 (stores: []) --> KSTREAM-KEY-SELECT-02 <--

Fwd: Exception in SFTP CSV SOURCE

Hi Guys By Trying SFTP CSV SOURCE i am getting this exception by using this configuration. what is the issue and how to resolve it? can anyone know? *Config:* { "name": "CsvSFTP1", "config": { "tasks.max": "1", "connector.class":

SFTP Connector

Hey Guys, Now i am trying to implement SFTP connector by using this configurations. i am using windows system. so i am having doubts like how to set a path? i tried to set like this in *config *as well as */mnt/c/users/vmuralidharan* but it doesn't work. so what i need to do ? {

Re: KTable as a compacted topic, implications

Hi Raffaele, hopefully others more knowledgeable will correct me if I'm wrong, but I don't believe anything gets read from the changelog topic. (other than at startup if the state-store needs to be restored) So in your Sub-topology-1, the only topic being consumed from is the repartition topic.

Add Jira ID to Contributors

Hi please add my jira id to Contributors my gira id is :ruanliang-hualun

Effect of changing rack id

Hey I would like to understand what is the effect of changing the rack ids of the brokers of an existing cluster. Let's say currently the brokers have rack values "even" and "odd". If I change the rack ids to "rack_1" and "rack_2" and do a rolling restart will there be any issues in the following

Persist Kafka Topics and ksqldb

After a *docker-compose restart. *All topics, ksqldb types are lost. Is there any way to safely persist all data ? docker-compse : --- version: "2" services: zookeeper: image: confluentinc/cp-zookeeper:latest container_name: zookeeper environment: ZOOKEEPER_CLIENT_PORT: 2181

Re: KTable as a compacted topic, implications

Thanks a lot Alex and Matthias, >From Alex answer, I understand that the record is written to the compacted topic as part of the transaction right ? On Tue, May 19, 2020 at 8:32 PM Matthias J. Sax wrote: > What Alex says is correct. > > The changelog topic is only written into during

Re: Persist Kafka Topics and ksqldb

You need to externalise your container data stores. Here's an example Docker Compose that does that: https://github.com/confluentinc/demo-scene/blob/master/wifi-fun/docker-compose.yml -- Robin Moffatt |

Re: Exception in SFTP CSV SOURCE

I forwarded wrongly I suppose.. So is there any solution for that above exception... On Tue, May 19, 2020, 22:35 Robin Moffatt wrote: > Hi Vishnu, > > I think there is a problem with your email client, it's just sent a > duplicate of each of your emails from yesterday? > > thanks, Robin. > >

Re: Exception in SFTP CSV SOURCE

Hi Vishnu, I think there is a problem with your email client, it's just sent a duplicate of each of your emails from yesterday? thanks, Robin. On Tue, 19 May 2020 at 16:44, vishnu murali wrote: > Hi Guys > > By Trying SFTP CSV SOURCE i am getting this exception by using this > configuration.

Re: KTable as a compacted topic, implications

What Alex says is correct. The changelog topic is only written into during processing -- in fact, you could consider this write a "side effect" of doing `store.put()`. The changelog topic is only read when recovering from an error and the store needs to be rebuilt from it. -Matthias On

Cassandra Sink Connector with KSQLDB : "DataException: Key must be a struct or map"

Hi there, I am having an issue with Cassandra connector : https://stackoverflow.com/questions/61897693/dataexception-key-must-be-a-struct-or-map Thank you so much for your help -- Mohammed Ait Haddou Linkedin.com/in/medait +212697937189

Re: KSQL to search for data in Kafka Topics

I'm not sure what you mean by "not using stream topology". What does that mean to you that you'd rather avoid? However, you can indeed use KSQL to define streams & tables that process data from a number of topics. However, I think you may have the misimpression that KSQL is designed so you can

Understanding Kafka controller log

Hi I am trying to understand the right way of viewing controller logs. As the controller logs are written in each broker, do we need to see all of them to know the state of the cluster at any given point in time ? To follow the leader of any given partition, do we need to correlate across

Re: KTable as a compacted topic, implications

Thanks a lot ! On Tue, May 19, 2020 at 10:40 PM Matthias J. Sax wrote: > Yes, for EOW, writing into changelog topics happens in the same > transaction as writing to output topic. > > You might be interesting in this blog post: >

Re: KTable as a compacted topic, implications

Yes, for EOW, writing into changelog topics happens in the same transaction as writing to output topic. You might be interesting in this blog post: https://www.confluent.io/blog/enabling-exactly-once-kafka-streams/ On 5/19/20 1:22 PM, Raffaele Esposito wrote: > Thanks a lot Alex and Matthias, >

Re: Understanding Kafka controller log

Hi Amitav, Brokers only write to controller log when they're the cluster controller. So if you wanted to see what the cluster controller was doing at a given point in time, then yep, you'd want to aggregate all controller.log files to handle controller changes. Typically controller status is very

Request for adding into contributors list

Hi, Please add my JIRA ID into the contributors list of Apache Kafka. Here is my JIRA profile: Username: adally Full name: jiamei xie Best Wishes, Jiamei IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are

Re: Kafka Connect Connector Tasks Uneven Division

Hi Please, can anybody help me with this? Regards and Thanks Deepak Raghav On Tue, May 19, 2020 at 1:37 PM Deepak Raghav wrote: > Hi Team > > We have two worker node in a cluster and 2 connector with having 10 tasks > each. > > Now, suppose if we have two kafka connect process W1(Port 8080)