Re: Can two different kafka clusters be used in kafka streams for consumption from 1 cluster and produce on another cluster

Hi Pushkar, unfortunately, cross cluster processing is currently not possible with Kafka Streams. Best, Bruno On 4/11/24 4:13 PM, Pushkar Deole wrote: Hi All, We are using a streams application and currently the application uses a common kafka cluster that is shared along with many other

Re: Messages disappearing from Kafka Streams topology

Hi, That sounds worrisome! Could you please provide us with a minimal example that shows the issue you describe? That would help a lot! Best, Bruno On 3/25/24 4:07 PM, Karsten Stöckmann wrote: Hi, are there circumstances that might lead to messages silently (i.e. without any logged

Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled

. Kind regards, Venkatesh From: Bruno Cadonna Date: Friday, 15 March 2024 at 8:47 PM To: users@kafka.apache.org Subject: Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled Hi Venkatesh, As you discovered, in Kafka Streams 3.5.1 there is no stop-the-world rebalancing. Static group

Re: Joining the Kafka Users Mailing List

Hi Vansh, Great that you want to join our community! Subscription to the mailing list is self-serve. See details to subscribe under the following link: https://kafka.apache.org/contact Thank you for your interest in Apache Kafka! Best, Bruno On 3/15/24 1:59 PM, Vansh Kabra wrote: Dear

Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled

I can use which can make offset lags observable even during rebalancing, can you please let me know? Thank you very much. Kind regards, Venkatesh From: Bruno Cadonna Date: Wednesday, 13 March 2024 at 8:29 PM To: users@kafka.apache.org Subject: Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app

Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled

Hi Venkatesh, Extending on what Matthias replied, a metadata refresh might trigger a rebalance if the metadata changed. However, a metadata refresh that does not show a change in the metadata will not trigger a rebalance. In this context, i.e., config METADATA_MAX_AGE_CONFIG, the metadata is

Re: [ANNOUNCE] Apache Kafka 3.7.0

, Arpit Goyal, Artem Livshits, Ashwin Pankaj, ashwinpankaj, atu-sharm, bachmanity1, Bob Barrett, Bruno Cadonna, Calvin Liu, Cerchie, chern, Chris Egerton, Christo Lolov, Colin Patrick McCabe, Colt McNealy, Crispin Bernier, David Arthur, David Jacot, David Mao, Deqi Hu, Dimitar Dimitrov, Divij Vaidya

Re: Question about using public classes from the internal packages

Hi John, In general, we do not guarantee anything on APIs of the internal package. That is also the reason why you do not need a KIP to change those classes. Any class for which the build generates Javadoc is considered public API [1]. For public APIs we guarantee backwards compatibility.

Re: Fwd: unsubscribe

Hi Clark, Unfortunately, I cannot verify your subscription but a mistake that often happens is that people try to unsubscribe from the mailing list with an e-mail address that is different from the one with which they subscribed to the mailing list. Please verify that you use the e-mail

Re: License Cost for Apache Kafka with Support

Hi Vinothkumar, The Apache Kafka project and the Apache Software Foundation do not sell licenses for Apache Kafka. Apache Kafka is distributed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0). This is an open source license that gives you quite some freedom.

Re: [ANNOUNCE] Apache Kafka 3.6.0

, Bruno Cadonna, Calvin Liu, Chaitanya Mukka, Chase Thomas, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton, Christo Lolov, Clay Johnson, Colin P. McCabe, Colt McNealy, d00791190, Damon Xie, Danica Fine, Daniel Scanteianu, Daniel Urban, David Arthur, David Jacot, David Mao, dengziming, Deqi Hu, Dimitar

Re: adding enum value in kafka streams

Hi Mariusz, How is fooKey de-/serialized? I ask that because maybe the serializer for fooKey cannot handle the extended enum. Best, Bruno On 9/20/23 12:22 AM, M M wrote: Hello, This is my first time asking a question on a mailing list, so please forgive me any inaccuracies. I am having a

Re: Kafka Streams, read standby time window store

(); /* */}/* and now it works well! thanks a lot for your help! On 9/6/23 16:05, Bruno Cadonna wrote: Hi Igor, Sorry to hear you have issues with querying standbys! I have two questions to clarify the situation: 1. Did you enable querying stale stores with StoreQueryParameters.fromNameAndType

Re: Kafka Streams, read standby time window store

Hi Igor, Sorry to hear you have issues with querying standbys! I have two questions to clarify the situation: 1. Did you enable querying stale stores with StoreQueryParameters.fromNameAndType(TABLE_NAME, queryableStoreType).enableStaleStores() as described in the blog post? 2. Since you

Re: Impact and fix of CVE-2023-34462 and CVE-2023-35116

Hi Sahil, we value that you raise security issues. However, since this is a sensitive area there are guidelines to follow as François pointed out. Additionally to what François said, I would like to remind you again (as I did in https://kafka.apache.org/project-security) to follow the

Re: Release plan required for version 3.5.1

Hi Sahil, Please have a look at the dependencies for Kafka 3.5.1: https://github.com/apache/kafka/blob/3.5.1/gradle/dependencies.gradle and compare it with your list of CVEs. Please also have a look here: https://kafka.apache.org/project-security If you discover a security issue please follow

Re: kafka streams re-partitioning on incoming events

e normally store in Processor in Topology? On Fri, Jul 14, 2023 at 3:50 PM Bruno Cadonna wrote: Hi Pushkar, you can use repartition() for repartition your data. Method through() is actually deprecated in favor of repartition(). Before you repartition you need to specify the new key with selectKe

Re: kafka streams re-partitioning on incoming events

Hi Pushkar, you can use repartition() for repartition your data. Method through() is actually deprecated in favor of repartition(). Before you repartition you need to specify the new key with selectKey(). What do you mean exactly with "...and then process them in that order"? The order of

Re: [ANNOUNCE] New committer: Greg Harris

Congratulations Greg! Best, Bruno On 10.07.23 17:47, Matthias J. Sax wrote: Congrats! On 7/10/23 8:45 AM, Chris Egerton wrote: Hi all, The PMC for Apache Kafka has invited Greg Harris to become a committer, and we are happy to announce that he has accepted! Greg has been contributing to

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.5.0

Mickael, Thanks for driving the release! Best, Bruno On 15.06.23 10:27, Mickael Maison wrote: The Apache Kafka community is pleased to announce the release for Apache Kafka 3.5.0. This is a minor release and it includes fixes and improvements from 201 JIRAs. All of the changes in this

Re: Kafka Stream: Cannot invoke "org.apache.kafka.streams.processor.internals.SourceNode.getTimestampExtractor()" because "source" is null

Hi Andrew, what version of Kafka Streams do you use? Since 2.7 there is a null check for the source node [1]. The following ticket might be related: https://issues.apache.org/jira/browse/KAFKA-10205 Best, Bruno [1]

Re: [VOTE] 3.4.1 RC3

Hi Prem, You need to remove yourself from the mailing list. Please find the details to do so here: https://kafka.apache.org/contact The interesting part for you is this: "To unsubscribe, send an email to users-unsubscr...@kafka.apache.org" Maybe you should also check if you are subscribed to

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

Congrats, David! Bruno On 10.03.23 01:36, Matthias J. Sax wrote: Congrats! On 3/9/23 2:59 PM, José Armando García Sancio wrote: Congrats David! On Thu, Mar 9, 2023 at 2:00 PM Kowshik Prakasam wrote: Congrats David! On Thu, Mar 9, 2023 at 12:09 PM Lucas Brutschy wrote:

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

Thank you all! I am really honoured to have become a PMC member. Looking forward to contributing further to Apache Kafka. Best, Bruno On 02.11.22 17:02, Levani Kokhreidze wrote: Congrats Bruno, well deserved! On Wed, 2 Nov 2022, 11:11 Jorge Esteban Quilcate Otoya, <

Re: Commercial support

/2022 11:05, Bruno Cadonna wrote: Hi Francois, I am not aware of such a page on the Apache Kafka website. There are a variety of companies that sell Kafka as a self-hosted platform or as a Cloud-hosted service. Those companies include Confluent (disclaimer: I work for them), Red Hat, AWS, Aiven

Re: Commercial support

Hi Francois, I am not aware of such a page on the Apache Kafka website. There are a variety of companies that sell Kafka as a self-hosted platform or as a Cloud-hosted service. Those companies include Confluent (disclaimer: I work for them), Red Hat, AWS, Aiven, Instaclustr, Cloudera, and

Re: Kafka stream moving from running to shutdown without any visible exception or error

Hi Samuel, from your few log messages I see that the global stream thread dies. That might be the cause of the shutdown. Look for what is causing your global stream thread/global store to die. Best, Bruno On 13.09.22 22:55, Samuel Azcona wrote: Hi guys, I have a kotlin App that run 5

Re: Apache atmosphere is sending an empty message every 5 minutes to the north bound. These messages contain a single byte, which is represented as a space character ' '. Is there any ability to disab

Hi Ori, I think you wrote your e-mail to the wrong list. This is the Apache Kafka user mailing list. Best, Bruno On 13.09.22 09:13, fcb man wrote: Hi Guys, We are using Apache Atmosphere and several users shared the same concern, where the north bound client (netcool in their case) is

Re: KAFKA 3.2.1 Vulnerabilities fix required

-Original Message- From: Bruno Cadonna Sent: 09 September 2022 03:15 PM To: users@kafka.apache.org Subject: Re: KAFKA 3.2.1 Vulnerabilities fix required Hi Sahil, I do not know when the next 3.2 patch release will be out. However, the upcoming 3.3.0 release fixes these vulnerabilities

Re: KAFKA 3.2.1 Vulnerabilities fix required

Hi Sahil, I do not know when the next 3.2 patch release will be out. However, the upcoming 3.3.0 release fixes these vulnerabilities. The release candidate 1 of the 3.3.0 release is currently in the voting phase. Best, Bruno On 09.09.22 11:33, Sahil Sharma D wrote: Hello team, We are

Re: [ANNOUNCE] Apache Kafka 3.2.1

, ING, LinkedIn, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 19 contributors to this release! Akhilesh Chaganti, Bruno Cadonna, Christopher L. Shannon, David Arthur, Divij Vaidya, Eugene Tolbakov, Guozhang Wang

Re: [ANNOUNCE] Apache Kafka 3.2.0

hould we fix that typo? Thanks! -James On May 17, 2022, at 9:01 AM, Bruno Cadonna wrote: The Apache Kafka community is pleased to announce the release for Apache Kafka 3.2.0 * log4j 1.x is replaced with reload4j (KAFKA-9366) * StandardAuthorizer for KRaft (KIP-801) * Send a hint to the part

Re: Leverage multiple disks for kafka streams stores

multiple volumes, sometimes it is cheaper to use several cheaper volumes in parallel than a single very expensive one. I personally believe that this should be considered for a KIP. Best regards, Adrian Tubio On Thu, May 19, 2022 at 1:49 PM Bruno Cadonna wrote: Hi Adrian, I am afraid that you

Re: Leverage multiple disks for kafka streams stores

Hi Adrian, I am afraid that you cannot set the state directory for a single state store to a different directory than all other stores. Maybe the following blog post can help you debug and solve your issue:

[ANNOUNCE] Apache Kafka 3.2.0

Khamphousone, bozhao12, Bruno Cadonna, Chang, Chia-Ping Tsai, Chris Egerton, Colin P. Mccabe, Colin Patrick McCabe, Cong Ding, David Arthur, David Jacot, David Mao, defhacks, dengziming, Ed B, Edwin, florin-akermann, GauthamM-official, GuoPhilipse, Guozhang Wang, Hao Li, Haoze Wu, Idan Kamara

Re: [kafka-clients] Re: [VOTE] 3.2.0 RC1

Jakub Scholz wrote: +1 (non-binding). I used the Scala 2.13 binaries and the staged Maven artifacts and ran various tests with them. Thanks for doing the release. Jakub On Tue, May 3, 2022 at 4:07 PM Bruno Cadonna wrote: Hello Kafka users, developers and client-developers, This is the second

[RESULTS] [VOTE] Release Kafka version 3.2.0

This vote passes with 5 +1 votes (3 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Bill Bejeck * Mickael Maison * David Jacot Committers: * Luke Chen Community: * Jakub Scholz 0 votes * No votes -1 votes * No votes Vote thread:

Re: [VOTE] 3.2.0 RC1

Hello Kafka users, developers and client-developers, I've got a green build with unit/integration tests: https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.2/45/ Best, Bruno On 03.05.22 16:07, Bruno Cadonna wrote: Hello Kafka users, developers and client-developers, This is the second

[VOTE] 3.2.0 RC1

Hello Kafka users, developers and client-developers, This is the second candidate for release of Apache Kafka 3.2.0. * log4j 1.x is replaced with reload4j (KAFKA-9366) * StandardAuthorizer for KRaft (KIP-801) * Send a hint to the partition leader to recover the partition (KIP-704) * Top-level

Re: [kafka-clients] Re: [VOTE] 3.2.0 RC0

fixes an issue introduced in 3.2.0 where in some of the error cases, the producer interceptor is called twice for the same record. Thanks, Jun On Tue, Apr 26, 2022 at 6:34 AM Bruno Cadonna <mailto:cado...@apache.org>> wrote: Hi all, This is a gently reminder to vot

Re: [VOTE] 3.2.0 RC0

://jenkins.confluent.io/job/system-test-kafka/job/3.2/24/ Thank you to Michal for voting on the release candidate. Best, Bruno On 15.04.22 21:05, Bruno Cadonna wrote: Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.2.0. * log4j 1.x is replaced

Re: Partnership: Apache Kafka & SantoDigital

Hi Amanda, Apache Kafka is an open source project. You can download, install, and use the software for free. Some companies offer Apache Kafka as a cloud service or as an on-prem software. You can find those companies by asking google. Best, Bruno On 25.04.22 17:11, Amanda Jampaulo

[VOTE] 3.2.0 RC0

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 3.2.0. * log4j 1.x is replaced with reload4j (KAFKA-9366) * StandardAuthorizer for KRaft (KIP-801) * Send a hint to the partition leader to recover the partition (KIP-704) * Top-level

Re: Need Help - getting vulnerability due to Log4j- v1.2.17 jar being used in Kafka_2.11-2.4.0.

Hi Sandip, I just merged the PR https://github.com/apache/kafka/pull/11743 that replaces log4j with reload4j. Reload4j will be part of Apache Kafka 3.2.0 and 3.1.1. Best, Bruno On 30.03.22 04:26, Luke Chen wrote: Hi Sandip, We plan to replace log4j with reload4j in v3.2.0 and v3.1.1.

Re: Setting up the CooperativeStickyAssignor in Java

Hi Richard, The group.instance.id config is orthogonal to the partition assignment strategy. The group.instance.id is used if you want to have static membership which is not related to the partition assignment strategy. If you think you found a bug, could you please open a JIRA ticket with

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.0.1

, Netflix, Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and Zalando, among others. A big thank you for the following 26 contributors to this release! A. Sophie Blee-Goldman, Andras Katona, Bruno Cadonna, Chris Egerton, Cong Ding, David Jacot, dengziming, Edoardo Comar, Ismael Juma

Re: Reducing issue

Hi Robin, since this seems to be a ksql question, you will more likely get an answer here: https://forum.confluent.io/c/ksqldb Best, Bruno On 01.02.22 10:03, Robin Helgelin wrote: Hi, Working on a small MVP and keep running into a dead end when it comes to reducing data. Began using

Re: Improving I/O on KafkaStreams

Hi Murilo, Have you checked out the following blog post on tuning performance of RocksDB state stores [1] especially the section on high disk I/O and write stalls [2]? Do you manage the off-heap memory used by RocksDB as described in the Streams docs [3]? I do not know what may have

Re: `java.lang.NoSuchFieldError: DEFAULT_SASL_ENABLED_MECHANISMS` after upgrading `Kafka-clients` from 2.5.0 to 3.0.0

Hi Bruce, I do not know the specific root cause of your errors but what I found is that Spring 2.7.x is compatible with clients 2.7.0 and 2.8.0, not with 3.0.0 and 2.8.1: https://spring.io/projects/spring-kafka Best. Bruno On 24.09.21 00:25, Chang Liu wrote: Hi Kafka users, I start

Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD

: Hi Bruno, thank you for your answer. I mean that the message that caused the exception was consumed and replaced thread will continue from the next message. How then does it handle uncaught exceptions, if it will fail again? On Tue, Aug 10, 2021 at 12:33 PM Bruno Cadonna wrote: Hi Yoda, What

Re: Kafka Streams Handling uncaught exceptions REPLACE_THREAD

Hi Yoda, What do you mean exactly with "skipping that failed message"? Do you mean a record consumed from a topic that caused an exception that killed the stream thread? If the record killed the stream thread due to an exception, for example, a deserialization exception, it will probably

Re: disappearing messages with kafka streams

Hi Günter, What is the timestamp of the records? The difference between the system time on the brokers and the record timestamp is used to decide whether a record segment should be removed because its retention time is exceeded. So if the retention time of the topic is set to 1.5 days, the

Re: command to get the value of a config from the broker

Hi Dhirendra, You could use the kafka-configs.sh script or in Java the admin client (see https://kafka.apache.org/28/javadoc/org/apache/kafka/clients/admin/Admin.html) Best, Bruno On 01.07.21 13:45, Dhirendra Singh wrote: Hi All, I want to get the value of a config from broker. I do not

Re: Issue with StreamsBuilder.stream(Pattern)

Hi Will, This looks like a bug to me. Could you please open a Jira with the stacktrace of the exception and a minimal repro example? Best, Bruno On 08.06.21 16:51, Will Bartlett wrote: Hi all, I'm hitting a NPE in a very basic repro. It happens when toString() is called on the

Re: Please add user heyingquan to the list of contributors

Hi, I added you to the list of contributors in the Apache Kafka JIRA project. You can now assign tickets to yourself. Welcome to Apache Kafka! Best, Bruno On 05.06.21 15:44, 和映泉 wrote: Please add user heyingquan to the list of contributors.

Re: Streams Partition Allocation Skewness

Hi Navneeth, I need some clarifications to be able to help you. First of all it would be useful to know if your topology is stateful, i.e., if it has to maintain state. Since you have two topics with 72 partitions but only 72 tasks (or partitions groups to assign) that need to be distributed

Re: State Store Data Retention

correct? Thanks On Mon, Apr 19, 2021 at 1:57 AM Bruno Cadonna wrote: Hi Upesh, The answers to your questions are: 1. The configs cleanup.policy and retention.ms are topic configs. Hence, they only affect the changelog of a state store, not the local state store in a Kafka Streams client

Re: Changing Replication Factor

Hi Marcus, 1. If you change REPLICATION_FACTOR_CONFIG without resetting the application (or deleting the changelog and repartition topics) and redeploy the Streams application, the replication factor of the internal topics will not change. The replication factor will only change for new

Re: Kafka Streams and retention limits

Additionally, with KIP-698 (https://cwiki.apache.org/confluence/x/7CnZCQ), we will add verifications of changelogs to find misconfigurations and report them to the users. Best, Bruno On 24.04.21 22:58, Matthias J. Sax wrote: The topics used by Kafka Streams to backup state stores, are

Re: State Store Data Retention

Hi Upesh, The answers to your questions are: 1. The configs cleanup.policy and retention.ms are topic configs. Hence, they only affect the changelog of a state store, not the local state store in a Kafka Streams client. Locally, window and session stores remove data they do not need

Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch

Congrats Randall! Well deserved! Bruno On 17.04.21 01:43, Matthias J. Sax wrote: Hi, It's my pleasure to announce that Randall Hauch in now a member of the Kafka PMC. Randall has been a Kafka committer since Feb 2019. He has remained active in the community since becoming a committer.

Re: [ANNOUNCE] New Committer: Bruno Cadonna

Thank you all for the kind words! Best, Bruno On 08.04.21 00:34, Guozhang Wang wrote: Hello all, I'm happy to announce that Bruno Cadonna has accepted his invitation to become an Apache Kafka committer. Bruno has been contributing to Kafka since Jan. 2019 and has made 99 commits and more

Re: [ANNOUNCE] New Kafka PMC Member: Bill Bejeck

Congrats Bill! Well deserved! Best, Bruno On 12.04.21 11:19, Satish Duggana wrote: Congratulations Bill!! On Thu, 8 Apr 2021 at 13:24, Tom Bentley wrote: Congratulations Bill! On Thu, Apr 8, 2021 at 2:36 AM Luke Chen wrote: Congratulations Bill! Luke On Thu, Apr 8, 2021 at 9:17 AM

Re: [kafka-clients] [VOTE] 2.7.1 RC0

Hi Mickael, Correction to my last e-mail: The bug does not break eos, but it breaks at-least-once. Bruno On 19.03.21 14:54, Bruno Cadonna wrote: Hi Mickael, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker

Re: [kafka-clients] [VOTE] 2.6.2 RC0

Hi Sophie, Correction to my last e-mail: The bug does not break eos, but it breaks at-least-once. Bruno On 19.03.21 14:54, Bruno Cadonna wrote: Hi Sophie, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker since

Re: [kafka-clients] [VOTE] 2.6.2 RC0

Hi Sophie, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker since the bug might break at-least-once and exactly-once processing guarantees. Feel free to set it back to major, if you think that it is not a

Re: [kafka-clients] [VOTE] 2.7.1 RC0

Hi Mickael, Please have a look at the following bug report: https://issues.apache.org/jira/browse/KAFKA-12508 I set its priority to blocker since the bug might break at-least-once and exactly-once processing guarantees. Feel free to set it back to major, if you think that it is not a

Re: [ANNOUNCE] New committer: Tom Bentley

Congrats, Tom! Best, Bruno On 15.03.21 18:59, Mickael Maison wrote: Hi all, The PMC for Apache Kafka has invited Tom Bentley as a committer, and we are excited to announce that he accepted! Tom first contributed to Apache Kafka in June 2017 and has been actively contributing since February

Re: Error upgrading KafkaStreams

2021 at 10:20, Bruno Cadonna wrote: Hi Murilo, A couple of questions: 1. What do you mean exactly with clean up? 2. Do you have acleanup policy specified on the changelog topics? Best, Bruno On 15.03.21 15:03, Murilo Tavares wrote: Hi Bruno No, I haven't tested resetting the application

Re: Redis as state store

Hi Alex, I guess wiping out the state directory is easier code-wise, faster, and/or at the time of development the developers did not design for remote state stores. But I do actually not know the exact reason. Off the top of my head, I do not know how to solve this for remote state stores.

Re: Error upgrading KafkaStreams

up and upgrade to 2.7. No error this time. Thanks Murilo On Mon, 15 Mar 2021 at 09:53, Bruno Cadonna wrote: Hi Murilo, Did you retry to upgrade again after you reset the application? Did it work? Best, Bruno On 15.03.21 14:26, Murilo Tavares wrote: Hi Bruno Thanks for your response. No, I

Re: Error upgrading KafkaStreams

Murilo On Mon, 15 Mar 2021 at 06:21, Bruno Cadonna wrote: Hi Murilo, No, you do not need any special procedure to upgrade from 2.4 to 2.7. What you see in the logs is not an error but a warning. It should not block you on startup forever. The warning says that the local states of task 7_17

Re: Redis as state store

Hi Alex, You are right! There is no "exactly once magic" backed into the RocksDB store code. The point is local vs remote. When a Kafka Streams client closes dirty under EOS, the state (i.e., the content of the state store) needs to be wiped out and to be re-created from scratch from the

Re: Error upgrading KafkaStreams

Hi Murilo, No, you do not need any special procedure to upgrade from 2.4 to 2.7. What you see in the logs is not an error but a warning. It should not block you on startup forever. The warning says that the local states of task 7_17 are corrupted because the offset you want to fetch of the

Re: RocksDB state store disk space estimation

Hi Chris, your estimation looks correct to me. I do not know how big M might be. Maybe the following link can help you with the estimation: https://github.com/facebook/rocksdb/wiki/Rocksdb-BlockBasedTable-Format There are also some additional files that RocksDB keeps in its directory. I

Re: kafka stream processor's process method

Hi Sathya, MyProcessor does not have access to MySource, because in MySource you just build the topology that is then executed by Kafka Streams. So you cannot send anything to MySource, because MyProcessor does not know anything about MySource. If you want to stop consumption upon an

Re: Kafka streams - how to handle application level exception in event processing

On Tue, 22 Sep 2020 at 08:12, Bruno Cadonna wrote: Hi Pushkar, I think there is a misunderstanding. If a consumer polls from a partition, it will always poll the next event independently whether the offset was committed or not. Committed offsets are used for fault tolerance, i.e., when

Re: Kafka streams - how to handle application level exception in event processing

streams internally? On Mon, Sep 21, 2020 at 9:01 PM Bruno Cadonna wrote: Hi Pushkar, If you want to keep the order, you could still use the state store I suggested in my previous e-mail and implement a queue on top of it. For that you need to put the events into the store with a key

Re: Kafka streams - how to handle application level exception in event processing

the other application starts up and required data becomes available in globalKtable On Mon, Sep 21, 2020 at 5:42 PM Bruno Cadonna wrote: Thank you for clarifying! Now, I think I understand. You could put events for which required data in the global table is not available into a state store

Re: kafka schema registry - some queries and questions

Hi Pushkar, This question is better suited for https://groups.google.com/g/confluent-platform since the Schema Registry is part of the Confluent Platform but not of Apache Kafka. Best, Bruno On 21.09.20 16:58, Pushkar Deole wrote: Hi All, Wanted to understand a bit more on the schema

Re: Kafka streams - how to handle application level exception in event processing

level error e.g. say, some data required for processing an input event is not available in the GlobalKTable since it is not yet synced with the global topic On Mon, Sep 21, 2020 at 4:54 PM Bruno Cadonna wrote: Hi Pushkar, Is the error you are talking about, one that is thrown by Kafka Streams

Re: Kafka streams - how to handle application level exception in event processing

Hi Pushkar, Is the error you are talking about, one that is thrown by Kafka Streams or by your application? If it is thrown by Kafka Streams, could you please post the error? I do not completely understand what you are trying to achieve, but maybe max.task.idle.ms [1] is the configuration

Re: Handle exception in kafka stream

Hi Deepak, Do you return DeserializationHandlerResponse.CONTINUE or DeserializationHandlerResponse.FAIL in your CustomExceptionHandler? With DeserializationHandlerResponse.CONTINUE, the processing of records should not stop and after the next offset commit the bad records should not be read

Re: Kafka streams sink outputs weird records

Hi Pirow, hard to to have an idea without seeing the code that is executed in the processors. Could you please post a minimal example that reproduces the issue? Best, Bruno On 20.08.20 14:53, Pirow Engelbrecht wrote: Hello, I’ve got Kafka Streams up and running with the following

Re: Sharing of State Stores

Hi Charles, Two transformers that share the same state store should end up into the same sub-topology. A sub-topology is executed by as many task as the number of partitions of the input topics. Each task processes the records from one input partition group (i.e. the same partition from both

Re: Question regarding Kafka Streams Global State Store

t; > Tel. +49 711 811-49893 > > ► Take a look: https://bgn.bosch.com/alias/bci > > > > -Ursprüngliche Nachricht- > Von: Bruno Cadonna > Gesendet: Dienstag, 19. Mai 2020 11:42 > An: Users > Betreff: Re: Question regarding Kafka Streams Global State Store > > Hi Georg, >

Re: Question regarding Kafka Streams Global State Store

> > Georg Schmidt-Dumont > BCI/ESW17 > Bosch Connected Industry > > Tel. +49 711 811-49893 > > ► Take a look: https://bgn.bosch.com/alias/bci > > > > -Ursprüngliche Nachricht- > Von: Bruno Cadonna > Gesendet: Dienstag, 19. Mai 2020 10:52 > An: Users

Re: Question regarding Kafka Streams Global State Store

Hi Georg, >From your description, I do not see why you need to use a global state instead of a local one. Are there any specific reasons for that? With a local state store you would have the previous record immediately available. Best, Bruno On Tue, May 19, 2020 at 10:23 AM Schmidt-Dumont Georg

Re: About Kafka Stream parallelism

Hi Rapeepat, 1. The parallelism of Kafka Streams does not only depend on the number of partitions of the input topic. It also depends on the structure of your topology. Your example topology topicA => transform1 => topicB => transform2 => topicC would be subdivided in two subtopologies: -

Re: KafkaStream groupBy + count on KTable behaviour

anism should be a > bit more transparent, but it aslo maybe that I'm plain wrong here :) > > Thanks ! > > On Thu, May 14, 2020 at 9:24 PM Bruno Cadonna wrote: > > > Hi Raffaele, > > > > Change is an internal class in Streams and also its SerDes are > > internal

Re: KafkaStream groupBy + count on KTable behaviour

gt; > Thanks, > > One more thing, As I told you I was consuming the repartitioning topic > > created by group by > > and I just saw the old and new value, as you are telling me now they are > > indeed marked as old and new, > > is this mark visible somehow consuming the

Re: KafkaStream groupBy + count on KTable behaviour

Hi Raffaele, In your example, Kafka Streams would send the new and the old value downstream. More specifically, the groupBy() would send (as you also observed) London, (old value: London, new value: null) Berlin, (old value: null, new value: Berlin) At the count() record London, (old value:

Re: Statestore restoration - Error while range compacting during restoring

Thank you, Nicolas! Bruno On Thu, Apr 16, 2020 at 2:24 PM Nicolas Carlot wrote: > > I've opened a Jira issue on the subject > https://issues.apache.org/jira/browse/KAFKA-9880 > > > Le jeu. 16 avr. 2020 à 13:14, Bruno Cadonna a écrit : > > > Hi Nicolas, > > &

Re: Statestore restoration - Error while range compacting during restoring

used." > > You meant "doesn't" right ? > > Ok, I will open a ticket, but I don't think my "fix" is the correct one. > Just ignoring the issue doesn't seem to be a correct solution :) > > Le jeu. 16 avr. 2020 à 11:49, Bruno Cadonna a écrit : >

Re: Statestore restoration - Error while range compacting during restoring

Hi Nicolas, Thank you for reporting this issue. As far as I understand, the issue is that bulk loading as done in Kafka Streams does work as expected if FIFO compaction is used. I would propose that you open a bug ticket. Please make sure to include steps to reproduce the issue in the ticket.

Re: KStream.groupByKey().aggregate() is always emitting record, even if nothing changed

o be > forwarded downstream*/).filter((k,v) -> return v !=null) > > Thanks > Sachin > > > On Tue, Feb 25, 2020 at 11:48 PM Bruno Cadonna wrote: > > > Hi Sachin, > > > > I am afraid I cannot follow your point. > > > > You can sti

Re: Error handling guarantees in Kafka Streams

Hi Magnus, with exactly-once, the producer commits the consumer offsets. Thus, if the producer is not able to successfully commit a transaction, no consumer offsets will be successfully committed, too. Best, Bruno On Wed, Feb 26, 2020 at 1:51 PM Reftel, Magnus wrote: > > Hi, > > From my

Re: KStream.groupByKey().aggregate() is always emitting record, even if nothing changed

unnecessarily cause data re-partitioning. Won't this be > in-efficient. > > Thanks > Sachin > > > > On Tue, Feb 25, 2020 at 10:52 PM Bruno Cadonna wrote: > > > Hello Guozhang and Adam, > > > > Regarding Guozhang's proposal please see recent discu

Re: KStream.groupByKey().aggregate() is always emitting record, even if nothing changed

Hello Guozhang and Adam, Regarding Guozhang's proposal please see recent discussions about `transformValues()` and returning `null` from the transformer:

Re: "Skipping record for expired segment" in InMemoryWindowStore

here that was only surfaced > > > > > through this warning. That said, maybe the metric is the more > > > appropriate > > > > > way to bring attention to this: not sure if it's info or debug level > > > > > though, or > > > > > how

  1   2   >