Re: Request to be added to kafka contributors list

Hi Matthia, I tried sign out and sign in, still can't find the "Assign" button, my JIRA ID is fanyan, could you help me set it again? Best, Fan From: Matthias J. Sax Sent: Saturday, May 18, 2024 4:06 To: users@kafka.apache.org Subject: Re: 回复: Request to be

Re: Fwd: Request to be added to kafka contributors list

Done. You should be all set :) -Matthias On 5/20/24 10:10 AM, bou...@ulukai.net wrote: Dear Apache Kafka Team,     I hope to post in the right place: my name is Franck LEDAY, under Apache-Jira ID "handfreezer".     I opened an issue as Improvement KAFKA-16707 but I failed to assigned

Re: Request for contributor list

What is your Jira ID? -Matthias On 5/20/24 9:55 AM, Brenden Deluna wrote: Hello, I am requesting to be added to the contributor list to take care of some tickets. Thank you.

Re: Release plan required

Zookeeper is already deprecated (since 3.5): https://kafka.apache.org/documentation/#zk_depr It's planned to be fully removed in 4.0 release. It's not confirmed yet, but there is a high probability that there won't be a 3.9 release, and that 4.0 will follow 3.8. -Matthias On 5/20/24 2:11

Fwd: Request to be added to kafka contributors list

Dear Apache Kafka Team,     I hope to post in the right place: my name is Franck LEDAY, under Apache-Jira ID "handfreezer".     I opened an issue as Improvement KAFKA-16707 but I failed to assigned it to me.     May I ask to be added to the contributors list for Apache Kafka? As I

Request for contributor list

Hello, I am requesting to be added to the contributor list to take care of some tickets. Thank you.

Kafka strange behaviors seen in logs

Hello! Currently I am running cluster of 3 kafka machines. Two of those are hosted in same data center and last one is in different. My kafka heap options are following: KAFKA_HEAP_OPTS=-Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20

RE: Release plan required

Hello, When Zookeeper is planned to depreciated from kafka, in which release this depreciation is planned? Regards, Sahil -Original Message- From: Sanskar Jhajharia Sent: Monday, May 20, 2024 1:38 PM To: users@kafka.apache.org Subject: Re: Release plan required [You don't often get

Re: Release plan required

Hey Sahil, You can find the complete details of the releases and bug fix releases here: https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan The next release in Pipeline currently is 3.8.0 ( https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0). There is al.o a

Release plan required

Hi team, We need the Kafka release plan for our Kafka upgrade planning, kindly share the latest Release Plan or when is the next release is planned and which version? Regards, Sahil

Re: broker crashing when running in raft mode

Seems that actually running ./bin/kafka-reassign-partitions.sh to first generate and then execute to move the topic is working. I later realized that in a cluster where the metadata have been migrated to k-raft, the following command: ./bin/kafka-topics.sh --describe --bootstrap-server

Re: 回复: Request to be added to kafka contributors list

Did you sign out and sign in again? On 5/17/24 9:49 AM, Yang Fan wrote: Thanks Matthias, I still can't find "Assign to me" button beside Assignee and Reporter. Could you help me set it again? Best regards, Fan 发件人: Matthias J. Sax 发送时间: 2024年5月17日 2:24 收件人:

broker crashing when running in raft mode

I'm trying to finish the migration of a dev cluster that used to use ZK. I finished the migration of Metadata from ZK to raft and it completed fairly fast. I did notice an error message: * java.lang.RuntimeException: The log dir

回复: Request to be added to kafka contributors list

Thanks Matthias, I still can't find "Assign to me" button beside Assignee and Reporter. Could you help me set it again? Best regards, Fan 发件人: Matthias J. Sax 发送时间: 2024年5月17日 2:24 收件人: users@kafka.apache.org 主题: Re: Request to be added to kafka contributors

Re: JMX Metrics UnderReplicatedPartitions And UnderMinIsrPartitionCount

Hi Artem, yes that's exactly the case! We're using Spring Kafka 2.7 and the @RetryableTopic has a default replication factor of 1 but our broker is configured with minIsr=2. In Spring Kafka 3 they changed it to -1 (meaning broker default value) [0]. Thanks! Andi [0]

Re: JMX Metrics UnderReplicatedPartitions And UnderMinIsrPartitionCount

Hi, this could be the case if you have a topic with min.insync.replicas > replication factor. On Fri, May 17, 2024 at 10:37 AM Andi Büchler wrote: > Hello, > > using the JmxTool I discovered that one can have _no_ under replicated > partitions, but have an under minISR partition count > 0.

JMX Metrics UnderReplicatedPartitions And UnderMinIsrPartitionCount

Hello, using the JmxTool  I discovered that one can have _no_ under replicated partitions, but have an under minISR partition count > 0. Does anyone know, how that can be? I was expecting that when a partition is below the minimum in sync replica count, it would be defined as under

Kafka Broker monitoring

Hi Team, We have some Splunk dashboards along with custom UI elements to report Kafka health status. We forward all Kafka health check statuses to be loaded into Splunk. However, we are encountering capacity issues on Splunk as we service multiple Kafka clusters across our data center as well

GetLogsDir API in Kafka 3.3.1 returns all topics even when topic name specified in args

Hello. I’m having a problem with Kafka protocol API. Requests: DescribeLogDirs Request (Version: 0) => [topics] topics => topic [partitions] topic => STRING partitions => INT32 My request contains `[{topic: “blah”, partitions: [0,1,2,3,4,5,6,7,8,9]}]`, but the result Responses:

Re: [ Questions on log4j file & version ]

Hi Ashok, Kafka 2.7.1 was built from the 2.7.1 tag [1] and looking at the dependencies in that version [2], it should have shipped with 1.2.17. You can verify this by looking for the log4j jar in your installation. Because of the security vulnerabilities you mention, Kafka switched to reload4j in

Re: Kafka streams stores key in multiple state store instances

Hello Kay, What you describe is "by design" -- unfortunately. The problem is, that when we build the `Topology` we don't know the partition count of the input topics, and thus, StreamsBuilder cannot insert a repartition topic for this case (we always assume that the partition count is the

Re: Request to be added to kafka contributors list

Thanks for reaching out Yang. You should be all set. -Matthias On 5/16/24 7:40 AM, Yang Fan wrote: Dear Apache Kafka Team, I hope this email finds you well. My name is Fan Yang, JIRA ID is fanyan, I kindly request to be added to the contributors list for Apache Kafka. Being part of this

Request to be added to kafka contributors list

Dear Apache Kafka Team, I hope this email finds you well. My name is Fan Yang, JIRA ID is fanyan, I kindly request to be added to the contributors list for Apache Kafka. Being part of this list would allow me to be assigned to JIRA tickets and work on them. Thank you for considering my

Large messages

Hi, I am having a bit of a problem here. It is a bit of an unusual use case. I am running a single node Kafka server and noticed that some messages I send are not passed through Kafka. They are silently lost. The topic is created but no data. Other topics on the same server work just fine.

[ Questions on log4j file & version ]

Hello Kafka Team, Request your help... We are using Apache Kafka kafka_2.13-2.7.1 & installed on a server. I understand it uses log4j java for logger purposes. But we don't know, what is the log4j version it is using? Recently we came to know that log4j_1.2.17 has some security issues, how to

Kafka streams stores key in multiple state store instances

Hi, we have a Kafka streams application which merges (merge, groupByKey, aggretgate) a few topics into one topic. The application is stateful, of course. There are currently six instances of the application running in parallel. We had an issue where one new Topic for aggregation did have

Re: Query regarding groupbykey in streams

If I read this correctly, your upstream producer which writes into the input topic of you KS app is using a custom partitioner? If you do a `groupByKey()` and change the key upstream, it would result in a repartition step, which would fall back to the default partioner. If you want to use a

Re: Kafka retention bug (?)

Thank you for the quick response! :) I've filed KAFKA-16779 to track the issue, with the information you requested. Please let me know if I can provide anything further. On Tue, May 14, 2024 at 8:28 PM Luke Chen wrote: > Hi Nicholas, > > I

Re: Kafka retention bug (?)

Hi Nicholas, I didn't know anything in v3.7.0 would cause this issue. It would be good if you could open a JIRA for it. Some info to be provided: 1. You said "in the past", what version of Kafka was it using? 2. What is your broker configuration? 3. KRaft mode? Combined mode? (controller + broker

Queries on Kafka

Hi all, We are facing bottlenecks related to 2 topics. I want to know where can we find assignment.partition.strategy in strimzi kafka. How to find how organizing and queuing load of logs coming into kafka. It looks all 3 partitions get hogged by source with tremendous amount of logs. The max

Re: Kafka streams state store return hostname as unavailable when calling queryMetadataForKey method

Ah. Well this isn't anything new then since it's been the case since 2.6, but the default task assignor in Kafka Streams will sometimes assign partitions unevenly for a time if it's trying to move around stateful tasks and there's no copy of that task's state on the local disk attached to the

Query regarding groupbykey in streams

Hi All, I have a custom partitioner to distribute the data across partitions in my cluster. My setup looks like below Version - 3.7.0 Kafka - 3 broker setup Partition count - 10 Stream server pods - 2 Stream threads in each pod - 10 Deployed in Kubernetes Custom partitioner on producer end. I

Re: Kafka Stream App Rolling Restarts - Too Many Rebalances Per Partition

Thank you, Sophie, for your reply and for these recommendations - they are informative. We are trying them out. Thanks, Nagendra U M From: Sophie Blee-Goldman Sent: Tuesday, May 7, 2024 1:54 AM To: users@kafka.apache.org Subject: Re: Kafka Stream App Rolling

Re: Kafka streams state store return hostname as unavailable when calling queryMetadataForKey method

Kafka upgraded from 3.5.1 to 3.7.0 version On Fri, May 10, 2024 at 2:13 AM Sophie Blee-Goldman wrote: > What version did you upgrade from? > > On Wed, May 8, 2024 at 10:32 PM Penumarthi Durga Prasad Chowdary < > prasad.penumar...@gmail.com> wrote: > > > Hi Team, > > I'm utilizing Kafka

Re: Kafka streams state store return hostname as unavailable when calling queryMetadataForKey method

What version did you upgrade from? On Wed, May 8, 2024 at 10:32 PM Penumarthi Durga Prasad Chowdary < prasad.penumar...@gmail.com> wrote: > Hi Team, > I'm utilizing Kafka Streams to handle data from Kafka topics, running > multiple instances with the same application ID. This enables

Kafka streams state store return hostname as unavailable when calling queryMetadataForKey method

Hi Team, I'm utilizing Kafka Streams to handle data from Kafka topics, running multiple instances with the same application ID. This enables distributed processing of Kafka data across these instances. Furthermore, I've implemented state stores with time windows and session windows. To retrieve

Re: Kafka Stream App Rolling Restarts - Too Many Rebalances Per Partition

Hey, Just skimming the config list, there are two things that immediately jumped out at me: 1. The default session timeout was bumped up to 45 seconds a little while ago. Not sure if you're overriding this or just using an older version, but I definitely recommend bumping this up to 45s.

Kafka Stream App Rolling Restarts - Too Many Rebalances Per Partition

Hi, We have multiple replicas of an application running on a kubernetes cluster. Each application instance runs a stateful kafka stream application with an in-memory state-store (backed by a changelog topic). All instances of the stream apps are members of the same consumer group.

few kafka topics got deleted post upgrade

Dear Kafka Community, We are upgrading our kafka instance from 2.8.1 version to 3.5.2. The setup we have was of 3 kafka broker nodes and 3 zookeeper nodes. We performed a rolling upgrade of the broker with the 3.5.2 version and observed that some topics got deleted post upgrade. we don't see

Re: How do we usually handle Node disconnected issue for kafka producer

Hi Kamal, I understand this, however the connections are maintained by a vertx Kafka client and I am not able to find a way to catch the closed connection and reopen it. Would setting the connections.max.idle.ms = -1 or max int/long help here. Thanks Sachin On Sat, 4 May 2024 at 11:06 PM,

Re: How do we usually handle Node disconnected issue for kafka producer

Hi Sachin, Why do you want to change the default settings? If the connection is open and unused, then it is fair to close the connection after the timeout and reopen it when required. On Fri, May 3, 2024 at 1:06 PM Sachin Mittal wrote: > Hi, > I am using a Kafka producer java client by vert.x

Re: Apache zookeeper going down every 168 hours

I mean, just applying the PR linked to the ticket ( https://github.com/apache/kafka/pull/6517) locally. Or just cloning the branch of this PR. Note that the patch is not merged to upstream yet so you should try it at your own risk. Thanks, 2024年5月4日(土) 15:52 Yogeshkumar Annadurai : > Hello, >

Re: Apache zookeeper going down every 168 hours

Hello, Thanks for your prompt response. How to apply patch for this? Could you please provide further more details? Regards Yogeshkumar A On Sat, 4 May 2024 at 9:33 AM, Haruki Okada wrote: > Thanks for sharing logs. > > Kafka has a mechanism to mark log dir as "failed" when IOException

Re: Apache zookeeper going down every 168 hours

Thanks for sharing logs. Kafka has a mechanism to mark log dir as "failed" when IOException happens on I/O operations, and it will shut down when all log dirs have marked as failed. (Kafka allows to set multiple log dirs for JBOD) >From server.log, we can see that the Kafka broker shut down

Re: Apache zookeeper going down every 168 hours

Hello, We see timeout error in server.log log files and properties files are attached for your reference regards Yogeshkumar A On Sat, May 4, 2024 at 5:27 AM Haruki Okada wrote: > Hi. > > log.retention shouldn't be related to the phenomenon. > Sounds like we should understand the situation

Re: Apache zookeeper going down every 168 hours

Hi. log.retention shouldn't be related to the phenomenon. Sounds like we should understand the situation more precisely to answer. > apache zookeeper connection is going down automatically How did you confirm this? On ZooKeeper log? Also, did you see any logs on Kafka side? (on stdout or

Apache zookeeper going down every 168 hours

Hello, We are using Apache kakfa in a development environment, where apache zookeeper connection is going down automatically every 168 hours. we observed that, log.retention.hours is set as 168 hours (7 days). I would like to understand the configuration for this kind of scenario (automatic

Re: Failed to initialize processor KSTREAM-AGGREGATE-0000000001

Can you file a ticket for it: https://issues.apache.org/jira/browse/KAFKA On 5/3/24 3:34 AM, Penumarthi Durga Prasad Chowdary wrote: Kafka versions 3.5.1 and 3.7.0, we're still encountering persistent issues. The Kafka Streams library is aligned with these Kafka versions. Upon analysis of the

Re: Failed to initialize processor KSTREAM-AGGREGATE-0000000001

Kafka versions 3.5.1 and 3.7.0, we're still encountering persistent issues. The Kafka Streams library is aligned with these Kafka versions. Upon analysis of the logs, it seems that the problem may occur when a Kafka node disconnects from Kafka Streams processes. This suspicion is supported by the

How do we usually handle Node disconnected issue for kafka producer

Hi, I am using a Kafka producer java client by vert.x framework. https://vertx.io/docs/apidocs/io/vertx/kafka/client/producer/KafkaProducer.html There is a producer setting in kafka: connections.max.idle.ms = 54 So if there are no records to produce then after 9 minutes I get this in my

Re: outerjoin not joining after window

How do you know this? First thing we do is write a log message in the value joiner. We don't see the log message for the missed records. Well, for left/right join results, the ValueJoiner would only be called when the window is closed... And for invalid input (or late record, ie, which

Re: outerjoin not joining after window

Hello, We did some testing in our test environment today. We are seeing some records processes where only one side of the join has a record. So that's good. However, we are still seeing some records get skipped. They never hit the value joiner (we write a log message first thing in the value

Re: outerjoin not joining after window

Thanks for the information. I ran the code using Kafka locally. After submitting some records inside and outside of the time window and grace, the join performed as expected when running locally. That gives some hope :) However, they never get into the join. How do you know this? Did

Re: outerjoin not joining after window

Matthias, Thanks for the information. I ran the code using Kafka locally. After submitting some records inside and outside of the time window and grace, the join performed as expected when running locally. I'm not sure why the join is not working as expected when running against our actual

Re: outerjoin not joining after window

I expect the join to execute after the 25 with one side of the join containing a record and the other being null Given that you also have a grace period of 5 minutes, the result will only be emitted after the grace-period passed and the window is closed (not when window end time is reached).

outerjoin not joining after window

Hello, I have a KStream to KStream outer join with a time difference of 25 minutes and 5 minutes of grace. When I get a record for one side of the join, but don't get a record on the other side of the join, I expect the join to execute after the 25 with one side of the join containing a record

Re: How to find out the end of the session window

Did you look into .windowedBy(...).emitStrategy(...) ? Using emit-final you would get an downstream even only after the window closed. -Matthias On 4/29/24 1:43 AM, Santhoshi Mekala wrote: Hi Team, We have the below requirement: We are processing batch logs in kstreams. Currently, we are

How to find out the end of the session window

Hi Team, We have the below requirement: We are processing batch logs in kstreams. Currently, we are storing the batch logs in kafka topic after processing. We would like to integrate with object storage to store the batch logs in object storage after processing. For batch logs, we are using

Re: [ANNOUNCE] New committer: Igor Soarez

Congrats, Igor! On Thu, Apr 25, 2024 at 6:10 AM Matthias J. Sax wrote: > Congrats! > > On 4/24/24 2:29 PM, Bill Bejeck wrote: > > Congrats Igor! > > > > -Bill > > > > On Wed, Apr 24, 2024 at 2:37 PM Tom Bentley wrote: > > > >> Congratulations Igor! > >> > >> On Thu, 25 Apr 2024 at 6:27 AM,

Re: [ANNOUNCE] New committer: Igor Soarez

Congrats! On 4/24/24 2:29 PM, Bill Bejeck wrote: Congrats Igor! -Bill On Wed, Apr 24, 2024 at 2:37 PM Tom Bentley wrote: Congratulations Igor! On Thu, 25 Apr 2024 at 6:27 AM, Chia-Ping Tsai wrote: Congratulations, Igor! you are one of the best Kafka developers!!! Mickael Maison 於

Re: Does the kafka controller protocol have a fallback?

Just following up, I realized I forgot to add some information. This is using kafka 3.5.1, I am in the process of setting up a kafka cluster which is configured to > use KRaft. There is a set of three controller nodes and a set of six > brokers. Both the controllers and the brokers are

KAFKA BROKER ISSUE

Dear Team Greetings for the day I have KAFKA running in a namespace on a k8s cluster exposed with NodePort. I took the backup of the whole namespace and restored it into a different namespace using velero. Now I have another kafka deployed on the 2nd namespace which is also exposed at some

Re: Kraft controller readiness checks

I’ll join Dima with the thanks, Luke. This seems to be indeed a good way of enforcing safe restarts. Thanks, Frank -- Francesco Burato | Software Development Engineer | Adobe | bur...@adobe.com | c. +44 747 9029370 From: Dima Brodsky Date: Monday, 22 April 2024

Re: Kraft controller readiness checks

Thanks Luke, this helps for our use case. It does not cover the buildout of a new cluster where there are no brokers, but that should be remedied by kip 919 which looks to be resolved in 3.7.0. ttyl Dima On Sun, Apr 21, 2024 at 9:06 PM Luke Chen wrote: > Hi Frank, > > About your question: >

Re: Kraft controller readiness checks

Hi Frank, About your question: > Unless this is already available but not well publicised in the documentation, ideally there should be protocol working on the controller ports that answers to operational questions like “are metadata partitions in sync?”, “has the current controller converged

Re: Failed to initialize processor KSTREAM-AGGREGATE-0000000001

Not sure either, but it sounds like a bug to me. Can you reproduce this reliably? What version are you using? It would be best if you could file a Jira ticket and we can take it from there. -Matthias On 4/21/24 5:38 PM, Penumarthi Durga Prasad Chowdary wrote: Hi , I have an issue in

Failed to initialize processor KSTREAM-AGGREGATE-0000000001

Hi , I have an issue in kafka-streams while constructing kafka-streams state store windows(TimeWindow and SessionWindow). While kafka-streams processing data sometimes intermittent kafka-streams process throwing below error ThreadName:

SQL Client fails with ClassNotFoundException when Hive dialect is chosen

Hi, I have *flink 1.19 *installation and start *sql-client locally*, like this: *./bin/sql-client.sh* then execute *SET 'table.sql-dialect' = 'hive'*; and have this exception: *[ERROR] Could not execute SQL statement. Reason:java.lang.ClassNotFoundException: org.antlr.runtime.tree.Tree* I'm

Re: Kraft controller readiness checks

Hi Luke, Thanks for the answers. I understand what you are describing in terms of rationale for using just the availability of the controller port to determine the readiness of the controller, but that is not fully satisfying under an operational perspective, at least based on the lack of

Does the kafka controller protocol have a backup?

Hello, I am in the process of setting up a kafka cluster which is configured to use KRaft. There is a set of three controller nodes and a set of six brokers. Both the controllers and the brokers are configured to use mTLS (Mutual TLS). So the part of the controller config looks like:

Re: Kraft controller readiness checks

Hello Frank, That's a good question. I think we all know there is no "correct" answer for this question. But I can share with you what our team did for it. Readiness: controller is listening on the controller.listener.names The rationale behind it is: 1. The last step for the controller node

Kraft controller readiness checks

Hello, I have a question regarding the deployment of Kafka using Kraft controllers in a Kubernetes environment. Our current Kafka cluster is deployed on K8S clusters as statefulsets without operators and our brokers are configured to use a script which marks the containers as unready if

Re: Streams group final result: EmitStrategy vs Suppressed

The main difference is the internal implementation. Semantically, both are equivalent. suppress() uses an in-memory buffer, while `emitStrategy()` does not, but modifies the upstream aggregation operator impl, and waits to send results downstream, and thus, it's RocksDB based. -Matthias

Re: Is there any recommendation about header max size?

I don't think that there is any specific recommendation. However, there is an overall max-message-size config that you need to keep in mind. -Matthias On 4/16/24 9:42 AM, Gabriel Giussi wrote: I have logic in my service to capture exceptions being thrown during message processing and produce

Re: [ANNOUNCE] New Kafka PMC Member: Greg Harris

Congrats Greg! On 4/15/24 10:44 AM, Hector Geraldino (BLOOMBERG/ 919 3RD A) wrote: Congrats! Well deserved From: d...@kafka.apache.org At: 04/13/24 14:42:22 UTC-4:00To: d...@kafka.apache.org Subject: [ANNOUNCE] New Kafka PMC Member: Greg Harris Hi all, Greg Harris has been a Kafka

Is there any recommendation about header max size?

I have logic in my service to capture exceptions being thrown during message processing and produce a new message to a different topic with information about the error. The idea is to leave the message unmodified, aka produce the exact same bytes to this new topic, therefore I'm planning on adding

UnwritableMetadataException on Startup

Hello, we currently have an "UnwritableMetadataException" error starting Kafka in our test environment. Unfortunately, I don't know whether it is due to an incorrect configuration or something similar. Kafka is running in a Docker Container, but I don't think that's the cause. When I Google this

Re: Streams group final result: EmitStrategy vs Suppressed

Hello, *[Not sure if my email went through as I was not subscribed to this mailing list. Here is my original email]* I found that there are two ways to send only the final result of a windowed groupBy, either using Suppressed

RE: [ANNOUNCE] Apache Kafka 3.6.2

On 2024/04/05 06:06:29 Manikumar wrote: > The Apache Kafka community is pleased to announce the release for > Apache Kafka 3.6.2 > > This is a bug fix release and it includes fixes and improvements from 28 JIRAs. > > All of the changes in this release can be found in the release notes: >

Streams group final result: EmitStrategy vs Suppressed

Hello, I found that there are two ways to send only the final result of a windowed groupBy, either using Suppressed .untilWindowCloses on the final KTable or

Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled

Thanks very much for your guidance, Matthias. Sorry for the delay in responding. I could see MSK (Kafka 3.6.0) broker log messages with “triggered followup rebalance scheduled for 0” but no occurrences of “no follow” in the broker logs. Hoping that rebalancing will not prevent the stalling

Would like your feedback on Apache Kafka based data pipeline platform: Braineous

Open Source High-Scale Data Pipeline Platform for Enterprise Data, Analytics, and Machine Learning Applications. Documentation: https://bugsbunnyshah.github.io/braineous/guides/developer-guide Get Started: https://bugsbunnyshah.github.io/braineous/get-started/ GitHub:

KRaft reconciling old default configs as dynamic configs

Hi there, I recently ran into an issue where I noticed that we had many old broker and configs set in zookeeper that were migrated over in KRaft mode. For example, this was a cluster default from 2020 that was moved over: get /config/brokers/

Re: Can two different kafka clusters be used in kafka streams for consumption from 1 cluster and produce on another cluster

Hi Pushkar, unfortunately, cross cluster processing is currently not possible with Kafka Streams. Best, Bruno On 4/11/24 4:13 PM, Pushkar Deole wrote: Hi All, We are using a streams application and currently the application uses a common kafka cluster that is shared along with many other

Can two different kafka clusters be used in kafka streams for consumption from 1 cluster and produce on another cluster

Hi All, We are using a streams application and currently the application uses a common kafka cluster that is shared along with many other applications. Our application consumes from topics that are populated by other applications and it consumes the events from those topics, processes those

Re: Messages disappearing from Kafka Streams topology

Hi Mangat, back to work now. I've configured out Streams applications to use exacly-once semantics, but to no avail. Actually, after some. more investigation I've come to suspect that the issue is somehow related to rebalancing. The initially shown topology lives inside a Quarkus Kafka Streams

Re: The impact of SSL encryption on Kafka transmission speed.

Thank for your help, It seems we have identified the issue. Kafka SSL is based on the javax.net.ssl implementation, which only supports single-threaded operations. During encryption, only one logical thread is working on the CPU. I am attempting to integrate other SSL implementations.

The impact of SSL encryption on Kafka transmission speed.

Hello everyone, After integrating SSL with Kafka, I noticed a significant decrease in producer speed. The speed of transmitting messages in plaintext is 101MB/s, which is exactly my network's maximum bandwidth speed. However, the speed of transmitting messages encrypted with SSL is only 19.17

kafka-streams stream-table join with a grace period does not respect passed serializer?

Hi, I'm trying to do a join between a stream and a KTable with a grace period, which requires a versioned table. However even if I specify a serializer everywhere, it seems this doesn't quite make it through to the RocksDBTimeOrderedKeyValueBuffer and I get an error when building/starting the

Update documentation SCRAM set up with Kraft

Hello Team, From https://cwiki.apache.org/confluence/display/KAFKA/KIP-900%3A+KRaft+kafka-storage.sh+API+additions+to+support+SCRAM+for+Kafka+Brokers I understand that SCRAM authentication is available for Kraft kafka clusters. However the official documentation is only referring to ZooKeeper

Re: [VOTE] 3.6.2 RC2

Hi Manikumar, I did the following checks: - downloaded and built from src - ran all unit test and integration test for clients - ran quickstart with Kraft mode - ran simple workloads with the console consumer/producer - checked all links All looks good to me with this. +1 (non-binding) Thanks!

Re: Fix slow processing rate in Kafka streams

Perf tuning is always tricky... 350 rec/sec sounds pretty low though. You would first need to figure out where the bottleneck is. Kafka Streams exposes all kind of metrics: https://kafka.apache.org/documentation/#kafka_streams_monitoring Might be good to inspect them as a first step -- maybe

Re: outerJoin confusion

I was able to get my test to complete correctly setting the internal setting and removing the calls to set the wall clock. props.put(InternalConfig.EMIT_INTERVAL_MS_KSTREAMS_OUTER_JOIN_SPURIOUS_RESULTS_FIX, 0L); Thanks Shashwat and Matthias for giving me two solutions. On Thu, Apr 4, 2024 at

[ANNOUNCE] Apache Kafka 3.6.2

The Apache Kafka community is pleased to announce the release for Apache Kafka 3.6.2 This is a bug fix release and it includes fixes and improvements from 28 JIRAs. All of the changes in this release can be found in the release notes: https://www.apache.org/dist/kafka/3.6.2/RELEASE_NOTES.html

Fix slow processing rate in Kafka streams

Hi All, My streams application is not processing more than 350 records/sec on a high load of 3milliom records produced every 2-3 minutes. My scenarios are as below - I am on Kafka and streams version of 3.5.1 . My key-value pair is in protobuf format . I do a groupbykey followed by TimeWindow of

Re: production ready for zookeeper to kraft migration

Matthieu, There are a few things to look out for during the migration. 1) Migration not starting. This is pretty much always going to be due to mis-configuration. Controller KRaftMigrationDriver logs should reveal what its waiting for 2) Migration of metadata fails. If some poison record in ZK

Re: outerJoin confusion

Yeah, that is some quirk of KS runtime... There is some internal config (for perf reasons) that delays emitting results... An alternative to advancing wall-clock time would be to set this internal config to zero, to disable the delay. Maybe we should disable this config when topology test

Re: production ready for zookeeper to kraft migration

Hey Luke, Thank you for the update. Out of curiosity if the migration is not working what are the symptoms ? is it just that the controller won't show that the migration is complete ? or could the controller claim (wrongfully) that the migration is complete when it's not ? Best. Matthieu On

Re: [EXTERNAL] Re: Kafka Streams 3.5.1 based app seems to get stalled

Glad to hear that the config change helped. For continuous rebalancing, it might be expected for KS, as KS uses the protocol in advanced ways. If you see log lines saying "follow up rebalance requested" than there is nothing to worry about, and the group is stable. If you see "no follow up

Re: [VOTE] 3.6.2 RC2

Thanks all for voting. I'm now closing the vote. The vote passes with - 3 +1 bindings votes from Divij Vaidya, Justine, and Manikumar - 3 +1 non-binding votes from Andrew, Jakub, and Lianet - 0 -1 votes I'll go ahead and finish the release process. Thanks, On Thu, Apr 4, 2024 at 8:53 PM

  1   2   3   4   5   6   7   8   9   10   >