Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

2023-02-06 Thread Greg Harris
: > > Hi Greg, > I actually just found the following comment on this PR for > https://issues.apache.org/jira/browse/KAFKA-13809: > https://github.com/apache/kafka/pull/12450 > > we get the same behavior (KAFKA-9228 notwithstanding) by passing the > original properties throug

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

2023-02-06 Thread Frank Grimes
Hi Greg, I actually just found the following comment on this PR for  https://issues.apache.org/jira/browse/KAFKA-13809:  https://github.com/apache/kafka/pull/12450 > we get the same behavior (KAFKA-9228 notwithstanding) by passing the original >properties through to tasks transparently It

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

2023-02-06 Thread Greg Harris
Frank, I think you're right that the KAFKA-9228 ticket doesn't capture every possible reconfiguration that might result in a dropped restart. The ticket calls out the FileStream connectors, which generate their configurations by dropping unknown config values, which is relatively uncommon

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

2023-02-06 Thread Frank Grimes
ht to only affect the following built-in connectors: FileStreamSourceConnector and the FileStreamSinkConnector see https://issues.apache.org/jira/browse/KAFKA-9228?focusedCommentId=16993990=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16993990 However, we are seeing

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

2023-02-03 Thread Greg Harris
rs at a higher level, when a worker is deciding whether to > write new task configs at all. > The relevant code is here: > https://github.com/apache/kafka/blob/6e2b86597d9cd7c8b2019cffb895522deb63c93a/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerde

Re: Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

2023-02-03 Thread Greg Harris
diagnostic purposes. What the Strimzi issue seems to describe (and probably the issue you are facing) occurs at a higher level, when a worker is deciding whether to write new task configs at all. The relevant code is here: https://github.com/apache/kafka/blob/6e2b86597d9cd7c8b2019cffb895522deb63c9

Kafka Connect ClusterConfigState.inconsistentConnectors() not handled by distributed Worker?

2023-02-03 Thread Frank Grimes
. In searching for others who have been bitten by this scenario we found the following thread on the Strimzi discussions pages: https://github.com/strimzi/strimzi-kafka-operator/discussions/7738 Both the symptoms and workaround described there match what we've seen.We've been doing some digging

Re: Some question about the kafka website language

2023-02-01 Thread LinuxGuy
n use English, not bad. > > > > > --原始邮件-- > 发件人: > "users" > < > xianshengzim...@gmail.com; > 发送时间:2023年2月1日

Re: Some question about the kafka website language

2023-01-31 Thread Nengda Ouyang
ndering even if you read document in your language, but > ultimately server.config and server.log will be in english only.. right? > > > > On Tue, 31 Jan 2023 at 9:00 PM, Nengda Ouyang > wrote: > > > I see the kafka website only have language version. Why doesn't provide > > other language. > > >

Re: Some question about the kafka website language

2023-01-31 Thread sunil chaudhari
I am just wondering even if you read document in your language, but ultimately server.config and server.log will be in english only.. right? On Tue, 31 Jan 2023 at 9:00 PM, Nengda Ouyang wrote: > I see the kafka website only have language version. Why doesn't provide > other language. >

Re: Some question about the kafka website language

2023-01-31 Thread Daniel Hinojosa
I can only assume that no one has come forward to translate. You can in most browsers, translate the page. In Chrome, you can right-click on the page and select "Translate to" and change to whatever language you need. It may not be perfect but it would get you going. Here is a snapshot (Not sure

Re: Some question about the kafka website language

2023-01-31 Thread Nengda Ouyang
I mean our have English website .Why doesn't provide other language. >

Some question about the kafka website language

2023-01-31 Thread Nengda Ouyang
I see the kafka website only have language version. Why doesn't provide other language.

Re: Spring-Kafka and maven enforcer plugin

2023-01-30 Thread Jan Hendriks
Nevermind, https://github.com/spring-projects/spring-kafka is the place to raise this. BR, Jan On Mon, Jan 30, 2023 at 1:33 PM Jan Hendriks wrote: > Hi, > we have issues with dependency convergence with Spring-Kafka-test and the > maven enforcer plugin. > A reproducer can be foun

Re: Trouble running kafka-server-start.sh from quickstart example

2023-01-30 Thread xiao cheng
i > wrote: > >> So did you provided server.properties properly? >> Check if property file exists at location. >> JAVA_HOME os availble? >> >> Regards, >> Sunil. >> >> On Mon, 30 Jan 2023 at 4:47 AM, xiao cheng >> wrote: >> >> &g

Fwd: Spring-Kafka and maven enforcer plugin

2023-01-30 Thread Jan Hendriks
Hi, we have issues with dependency convergence with Spring-Kafka-test and the maven enforcer plugin. A reproducer can be found at https://gist.github.com/DaHoC/67daf9a7cb90e8626dbe6a144e93fa16 - simply run "mvn clean install" in the folder of the provided pom.xml The divergences we

Spring-Kafka and maven enforcer plugin

2023-01-30 Thread Jan Hendriks
Hi, we have issues with dependency convergence with Spring-Kafka-test and the maven enforcer plugin. A reproducer can be found at https://gist.github.com/DaHoC/67daf9a7cb90e8626dbe6a144e93fa16 - simply run "mvn clean install" in the folder of the provided pom.xml The divergences we

Re: Trouble running kafka-server-start.sh from quickstart example

2023-01-30 Thread xiao cheng
/Home On Mon, Jan 30, 2023, 6:23 AM sunil chaudhari wrote: > So did you provided server.properties properly? > Check if property file exists at location. > JAVA_HOME os availble? > > Regards, > Sunil. > > On Mon, 30 Jan 2023 at 4:47 AM, xiao cheng > wrote: > >

Re: Trouble running kafka-server-start.sh from quickstart example

2023-01-29 Thread sunil chaudhari
So did you provided server.properties properly? Check if property file exists at location. JAVA_HOME os availble? Regards, Sunil. On Mon, 30 Jan 2023 at 4:47 AM, xiao cheng wrote: > Hi Sunil, > bin/kafka-server-start.sh --help give me output of > > bin/kafka-server-start.sh --help

Re: Trouble running kafka-server-start.sh from quickstart example

2023-01-29 Thread xiao cheng
Hi Sunil, bin/kafka-server-start.sh --help give me output of bin/kafka-server-start.sh --help [2023-01-30 00:14:32,122] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) USAGE: java [options] kafka.Kafka server.properties [--override property

Re: Trouble running kafka-server-start.sh from quickstart example

2023-01-27 Thread sunil chaudhari
Please try executing.. bin/kafka-server-start.sh —help On Fri, 27 Jan 2023 at 8:47 PM, xiao cheng wrote: > Hey all, > > I recently tried to run kafka locally on my macos again after a while (it > used to work). I followed the quickstart guide from > https://kafka.apache

Apache Ranger and Kafka compatibility

2023-01-27 Thread Doug Whitfield
I can look at the pom files and see the recommended versions of Kafka with Ranger. However, I would like to upgrade Kafka. Does anyone have any experience upgrading Kafka but keeping Ranger on an older version, say 1.2? Best Regards, Douglas Whitfield | Enterprise Architect, OpenLogic<ht

Re: Kafka integration with Legacy Systems (OCI)

2023-01-27 Thread andreasvdber...@avathar.be
hi, there is a kafka connector for weblogic. https://docs.confluent.io/kafka-connectors/weblogic-source/current/overview.html <https://docs.confluent.io/kafka-connectors/weblogic-source/current/overview.html> greetings, Andreas > Op 27 jan. 2023, om 16:55 heeft Olamide

Kafka integration with Legacy Systems (OCI)

2023-01-27 Thread Olamide Sode
Hi, I am looking for an architecture that uses kafka to integrate with an Oracle Cloud Infrastructure Legacy system that runs a weblogic server as the application server. Would be great if I can be pointed to a starting guide or direction? Thank you. Ola

Trouble running kafka-server-start.sh from quickstart example

2023-01-27 Thread xiao cheng
Hey all, I recently tried to run kafka locally on my macos again after a while (it used to work). I followed the quickstart guide from https://kafka.apache.org/quickstart. However, when I try to run it (bin/kafka-server-start.sh config/server.properties) again today I am seeing error message

Re: [ANNOUNCE] Apache Kafka 3.3.2

2023-01-25 Thread Bill Bejeck
Thanks for running the release Chris! On Mon, Jan 23, 2023 at 5:18 PM Chris Egerton wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 3.3.2 > > Apache Kafka 3.3.2 is a bugfix release and it contains, among other things, > fixes for 20

Kafka Topic partitions losing replication factor

2023-01-23 Thread Dima Brodsky
Hello, I was wondering if anybody has seen the following: I have a topic with about 200 partitions and with a replication factor of 3. Once in a while I am seeing a partition or two lose a replica. That is the replica list for those partitions goes from 3 to 2 brokers. Kafka does not see

Re: Custom Kafka Streams State Restore Logic

2023-01-23 Thread Upesh Desai
and is performant enough thus far! Thanks for your help. Cheers, Upesh Upesh Desai | Senior Software Developer | ude...@itrsgroup.com www.itrsgroup.com From: Matthias J. Sax Date: Monday, January 23, 2023 at 8:05 PM To: users@kafka.apache.org Subject: Re: Custom Kafka Streams State Restore Logic

Re: Custom Kafka Streams State Restore Logic

2023-01-23 Thread Matthias J. Sax
) create a custom state store from scratch, or 2) see if there is a post-restore hook that can then call a method to scan the whole store and build up the time bucket map before starting to process. Any advice on Kafka streams / state store logic would be appreciated! -Upesh Upesh Desai

Kafka Broker (with no leader partitions) : JVM heap usage fluctuates wildly in less than a minute.

2023-01-23 Thread Neeraj Vaidya
Hi All,I've posted this question on SO about JVM heap usage wildly fluctuating for a Broker which has no leader partitions.If anyone has a clue, please assist. Kafka Broker JVM heap usage fluctuates wildly | | | | | | | | | | | Kafka Broker JVM heap usage fluctuates wildly I have 8

[ANNOUNCE] Apache Kafka 3.3.2

2023-01-23 Thread Chris Egerton
The Apache Kafka community is pleased to announce the release for Apache Kafka 3.3.2 Apache Kafka 3.3.2 is a bugfix release and it contains, among other things, fixes for 20 issues reported since 3.3.1. All of the changes in this release can be found in the release notes: https://www.apache.org

Re: Kafka issues related 3.2.3 version zookeeper less cluster

2023-01-23 Thread Kunal Jadhav
M Kunal Jadhav > wrote: > > > Hello All, > > > > We are facing some issues in the 3.2.3 kafka version cluster. We have > > implemented 3 brokers cluster on a single node server in the kubernetes > > environment. After every week are facing issues which are

Re: Custom Kafka Streams State Restore Logic

2023-01-18 Thread Upesh Desai
starting to process. Any advice on Kafka streams / state store logic would be appreciated! -Upesh Upesh Desai | Senior Software Developer | ude...@itrsgroup.com www.itrsgroup.com From: Matthias J. Sax Date: Wednesday, January 18, 2023 at 12:50 AM To: users@kafka.apache.org Subject: Re: Custom

Re: Custom Kafka Streams State Restore Logic

2023-01-17 Thread Matthias J. Sax
Guess it depends what you actually want to achieve? Also note: `InMemoryWindowStore` is an internal class, and thus might change at any point, and it was never designed to be extended... -Matthias On 1/13/23 2:55 PM, Upesh Desai wrote: Hello all, I am currently working on creating a new

Re: Kafka issues related 3.2.3 version zookeeper less cluster

2023-01-16 Thread Divij Vaidya
are correlated with this time frame? Thanks! Regards, Divij Vaidya On Thu, Jan 12, 2023 at 3:13 PM Kunal Jadhav wrote: > Hello All, > > We are facing some issues in the 3.2.3 kafka version cluster. We have > implemented 3 brokers cluster on a single node server in the kubernetes > environme

Custom Kafka Streams State Restore Logic

2023-01-13 Thread Upesh Desai
Hello all, I am currently working on creating a new InMemoryWindowStore, by extending the default in memory window store. One of the roadblocks I’ve run into is finding a way to add some custom logic when the state store is being restored from the changelog. I know that this is possible if I

Subscribe to Kafka users mailing list

2023-01-13 Thread 华科-许加耀
subscribe

Kafka issues related 3.2.3 version zookeeper less cluster

2023-01-12 Thread Kunal Jadhav
Hello All, We are facing some issues in the 3.2.3 kafka version cluster. We have implemented 3 brokers cluster on a single node server in the kubernetes environment. After every week are facing issues which are listed below. So, please help us to resolve these issues. Thanks to all in advance

[RESULTS] [VOTE] Release Apache Kafka version 3.3.2

2023-01-11 Thread Chris Egerton
This vote passes with 8 +1 votes (3 bindings) and no 0 or -1 votes. +1 votes PMC Members: * Manikumar Reddy * Mickael Maison * Tom Bentley Committers: * Satish Duggana Community: * Yash Mayya * Divij Vaidya * Jakub Scholz * Federico Valeri 0 votes * No votes -1 votes * No votes Vote thread:

Kafka on GKE - automating distribution of TLS certs to external Kafka clients on certs expiry

2023-01-05 Thread karan alang
Hello All, We have a Strimzi kafka cluster on GKE with TLS implemented, the TLS certs have a default expiry of 1 year i.e. the certs get renewed every year Currently, on expiry - we have to provide the external Kafka clients with new certs, the objective is to automate this process i.e. One

Re: Kafka Stream: The state store, wkstore, may have migrated to another instance

2023-01-02 Thread Nawal Sah
org.apache.kafka.streams.state.internals.WrappingStoreProvider.stores(WrappingStoreProvider.java:67) > >* > > *at > > > org.apache.kafka.streams.state.internals.CompositeReadOnlyWindowStore.fetchAll(CompositeReadOnlyWindowStore.java:175)* > > > > To fix this, I must restart all the pod

Re: Kafka Stream: The state store, wkstore, may have migrated to another instance

2022-12-29 Thread Matthias J. Sax
vider.java:67) * *at org.apache.kafka.streams.state.internals.CompositeReadOnlyWindowStore.fetchAll(CompositeReadOnlyWindowStore.java:175)* To fix this, I must restart all the pods/instances or the cluster. *Steps to reproduce* 1. Kafka Stream application environment 'kafka-streams', versi

Kafka Stream: The state store, wkstore, may have migrated to another instance

2022-12-29 Thread Nawal Sah
e.fetchAll(CompositeReadOnlyWindowStore.java:175)* To fix this, I must restart all the pods/instances or the cluster. *Steps to reproduce* 1. Kafka Stream application environment 'kafka-streams', version: '3.0.1' 'kafka-clients', version: '3.0.1' 2. Create a cluster environment with at least two replicati

Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.

2022-12-22 Thread Divij Vaidya
nity. > > -- > Divij Vaidya > > On Mon, Dec 19, 2022 at 1:23 PM Atul Kumar (atkumar3) > wrote: > >> Hi Divij, >> >> As this issue manifests after the Kafka controller restart, is there a >> way to detect this corrupted Kafka clust

Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.

2022-12-22 Thread Divij Vaidya
sts after the Kafka controller restart, is there a > way to detect this corrupted Kafka cluster before restarting the controller? > > > > > > Regards,​ > Atul > > From: Atul Kumar (atkumar3) > Sent: 16 December 2022 18:24 > To:

Delete Kafka topics replicated by mirror maker 2

2022-12-21 Thread Shirolkar, Anup
Hi Team, I have configured two Kafka clusters with 2-way replication using MirrorMaker2 (mm2). I want to delete a topic from source cluster and its replica from destination cluster. When I delete a topic in the source cluster and then delete the replica in the destination cluster

Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.

2022-12-19 Thread Atul Kumar (atkumar3)
Hi Divij, As this issue manifests after the Kafka controller restart, is there a way to detect this corrupted Kafka cluster before restarting the controller? Regards,​ Atul From: Atul Kumar (atkumar3) Sent: 16 December 2022 18:24 To: atul.ku

Re: [SUSPECTED SPAM] Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.

2022-12-16 Thread Atul Kumar (atkumar3)
Hi Divij, Thanks for the reply! We were using the Kafka-manager tool (with an older Kafka client) which was the root cause of this issue. Regards,​ Atul From: Divij Vaidya Sent: 08 December 2022 17:55 To: atul.ku...@appdynamics.com.invalid Cc: users

Is there an option to ignore the apiVersion between Kafka versions

2022-12-15 Thread Swathi Mocharla
Hello, We are trying to upgrade between 2 versions of kafka where the apiVersions of multiple apis have changed. During upgrade we notice a number of UnsupportedVersionExceptions which we want to ignore (during upgrade). Few questions: 1. Is it ok to ignore mismatch of apiVersions? 2

How to use a kafka container in kubernetes?

2022-12-12 Thread Mario Ricci
://stackoverflow.com/questions/74776463/kafka-container-on-local-kubernetes-for-integration-tests-crashing-or-not-acce Any help would be much appreciated, *Mario Ricci* Technical Lead Architect | D Construction 10368 Westmoor Dr., Westminster, CO

Kafka connect SMT does not work for primary key columns

2022-12-11 Thread Fuxin Hao
t", "database.server.name": "test", "slot.name" : "test", "key.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter.schemas.enabled": true, "value.converter": "org.apache.kafka.connect.js

Re: same keys appearing in state stores on different pods when using branches in kafka streams

2022-12-08 Thread Pushkar Deole
t, even > >>> though both branches use the same grouping logic, Streams can’t detect > that > >>> they are the same. It just sees two group-bys and therefore introduces > two > >>> repartitions, with a separate downstream task for each. > >>&g

Re: Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.

2022-12-08 Thread Divij Vaidya
Hi Atul There is a known bug which has similar symptoms that you observed. See: https://issues.apache.org/jira/browse/KAFKA-14190 The bug manifests when you use a client < 2.8.0 with `--zookeeper` flag. It could be avoided by either upgrading the client or by using `--bootstrap-server` inst

Critical bug in Kafka 2.8.1 | topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.

2022-12-08 Thread Atul Kumar (atkumar3)
Hello everyone, We are using Kafka 2.8.1, and recently while rolling restart of one of our clusters, we encountered the following issue on the Kafka cluster topic Id in memory: <> does not match the topic Id for partition <> provided in the request: <>.  All th

Kafka Protocol & Collision/Replay Attacks

2022-12-08 Thread Ciaran McHugh
I recently created a Threat Model for an application I am working on using Microsoft's Threat Modelling Tool<https://learn.microsoft.com/en-gb/azure/security/develop/threat-modeling-tool>. It highlight two potential threats in relation to the use of the Kafka Protocol<https://kafka.a

Kafka Consumer - How to set fetch.max.bytes higher than the default 50mb?

2022-12-06 Thread Rama Eshel
I want my consumers to process large batches, so I aim to have the consumer listener "awake", say, on 1800mb of data or every 5min, whichever comes first. Mine is a kafka-springboot application, the topic has 28 partitions, and this is the configuration I explicitly change: |

Re: same keys appearing in state stores on different pods when using branches in kafka streams

2022-12-05 Thread John Roesler
the same. It just sees two group-bys and therefore introduces two >>> repartitions, with a separate downstream task for each. >>> >>> You might want to print out the topology description and visualize it >>> with https://zz85.github.io/kafka-streams-viz/ . That will sho

Re: Supported Kafka/Zookeeper Version with ELK 8.4.3

2022-12-05 Thread sunil chaudhari
Hi All, Well this question can be asked on discuss.elastic.co However I would like to answer this. I am sure you need compatibility with logstash. You can go through this document for logstash-kafka integration https://www.elastic.co/guide/en/logstash/current/plugins-integrations-kafka.html https

Re: Supported Kafka/Zookeeper Version with ELK 8.4.3

2022-12-05 Thread Colin McCabe
Hi, Sorry, we do not develop ELK. In fact, I'm not sure what that acronym refers to. I would suggest checking in with support for that product / project, since it is not part of Apache Kafka. best, Colin On Fri, Oct 28, 2022, at 06:23, Kumar, Sudip wrote: > Hi Team, > > We

Re: same keys appearing in state stores on different pods when using branches in kafka streams

2022-12-05 Thread Pushkar Deole
>> they are the same. It just sees two group-bys and therefore introduces two >> repartitions, with a separate downstream task for each. >> >> You might want to print out the topology description and visualize it >> with https://zz85.github.io/kafka-streams-viz/ . That wil

Re: Kafka service unavailable

2022-12-05 Thread John Roesler
Hi Rakesh, I'm sorry for your trouble. The mailing list doesn't transmit embedded images, so we can't see the information you provided. Maybe you can create a Github Gist or open a Jira ticket at https://issues.apache.org/jira/projects/KAFKA/issues ? Thanks, -John On Sun, Dec 4, 2022, at 23

Kafka service unavailable

2022-12-04 Thread Rakesh Kumar Tunuguntla
Hi Team, We have a Kafka service installed and we are unable to restore the service or restart it. When we check the status, it is showing as failed. [cid:image001.png@01D9089C.D40CA860] Can you please look into the query and provide the resolution faster. Warm Regards, Rakesh Kumar

RE: Kafka support for TLSv1.3

2022-12-02 Thread Deepak Nangia
Hi Ismael, Thanks for the quick response. Are there any plans in future anytime soon to support latest versions of Kafka over TLSv1.3 with Java8u261 and above? If yes, by when can we expect that. Regards Deepak Nangia From: Ismael Juma Sent: 02 December 2022 23:27 To: users@kafka.apache.org

Re: Kafka support for TLSv1.3

2022-12-02 Thread Ismael Juma
Hi Deepak, Kafka currently only supports TLS 1.3 with Java 11 or newer. As you said, Kafka 2.6 added support for it, but we recommend using something more recent as there were some fixes after the original release. Ismael On Fri, Dec 2, 2022 at 7:21 AM Deepak Nangia wrote: > Hello All, &g

Kafka support for TLSv1.3

2022-12-02 Thread Deepak Nangia
Hello All, I have few queries regarding Kafka support for TLSv1.3 as below: 1. Which Kafka version and above support TLSv1.3 * In latest Kafka release notes, I see that TLSv1.3 has been introduced in Kafka 2.6.0 and above. Reference<https://kafka.apache.org/documentat

Kafka log segments recovery/replica creation on startup

2022-12-01 Thread Mcs Vemuri
Hello, Can anyone please point to any docs re how replica recovery works when broker is started?  The documentation has information about broker rebuilding based on data on data dir- but what happens if data is completely lost? Would an empty replica be created based on zookeeper state- and if

[ANNOUNCE] Call for papers: Kafka Summit London 2023

2022-11-30 Thread Chris Egerton
Hi everyone, The call for papers (https://sessionize.com/kafka-summit-london-2023/) is now open for Kafka Summit London 2023, and you are all welcome to submit a talk. We are looking for the most interesting, most informative, most advanced, and most generally applicable talks on Apache Kafka

Re: same keys appearing in state stores on different pods when using branches in kafka streams

2022-11-28 Thread Pushkar Deole
e it with > https://zz85.github.io/kafka-streams-viz/ . That will show you whether > the stores wind up in the same task or not. > > The visualization will also show you the names of the input topics for > those two partitions, which you can use in conjunction with the metadata > meth

Re: Granting permission for Create KIP and contribute to kafka

2022-11-27 Thread Luke Chen
Hi t-mac, I've granted your JIRA account. But I can't find the wiki id: "ws" in the wiki system. Are your sure you registered with "ws" here <https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals>? Any more info you can provide to us, like acc

Re: Stream sinks are not constructed when application starts up before Kafka broker

2022-11-23 Thread John Roesler
the topology, or are you calling the Kafka Streams library directly to build the topology? If the problem is that the sink node is missing completely, it’s hard to imagine how the problem could be Streams. When you are building the topology in Streams, there is no connection to Kafka at all

Kafka 4.x release date

2022-11-23 Thread Moravcik . Juraj
Good morning, we are using for our product (Fare Collection System Ticketing) Apache Kafka software. https://www.scheidt-bachmann.de/en/fare-collection-systems/home One of task is update logging from version1 to version2. https://issues.apache.org/jira/browse/KAFKA-9366 Based on this link

Re: same keys appearing in state stores on different pods when using branches in kafka streams

2022-11-23 Thread John Roesler
might want to print out the topology description and visualize it with https://zz85.github.io/kafka-streams-viz/ . That will show you whether the stores wind up in the same task or not. The visualization will also show you the names of the input topics for those two partitions, which you can use

Active <> Active MirrorMaker2 setup via dedicated Kafka Connect cluster

2022-11-23 Thread Sriram Ganesh
Hi, I am trying to set up active <> active mm2 via Kafka connect distributed cluster. It seems not possible because of the limitations like *bootstrap.servers *property. And also as per this KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMa

same keys appearing in state stores on different pods when using branches in kafka streams

2022-11-23 Thread Pushkar Deole
Hi All, I have a stream application that creates 2 branches. Each branch includes a state store where the status field of the kafka message determines the branch, and therefore the state store used: Status OPEN = State store name totals Status CLOSED = State store name records I’m seeing

Stream sinks are not constructed when application starts up before Kafka broker

2022-11-22 Thread Alexander Kau
My team is building a set of services that use Kafka Connect and Debezium to forward data changes from our Postgres database to Kafka, and then use Kafka Streams (via Spring Cloud Stream) to process this data and output an aggregate of the source data. We have been trying to track down an issue

Can not start python kafka consumer using SASL/SCRAM

2022-11-21 Thread Huy Nguyen
I'm having an issue with Kafka. Whenever the consumer use SASL_SSL or SASL_PLAINTEXT, it can't start. I've tried to change the consumer to use PLAINTEXT or SSL (without SASL), and its working fine Below are my configurations and error: *config/server.properties:* listeners=PLAINTEXT://localhost

Re: Kafka Streams possible partitioner bug

2022-11-20 Thread Upesh Desai
: Kafka Streams possible partitioner bug Hey Upesh, are you trying to plug in the custom partitioner via the `partitioner.class` ProducerConfig? That won't work in Streams for the exact reason you highlighted, which is why Streams has its own version of the interface called StreamPartitioner

Re: Kafka 3.3.1, unable to start services in cygwin

2022-11-20 Thread sunil chaudhari
3.3.1.tgz > > and am unable to start services from a cygwin bash shell on my desktop > using zookeeper. Relevant error > > $ zookeeper-server-start.sh $KAFKA/config/zookeeper.properties > [0.003s][error][logging] Invalid decorator > '/cygwin64/usr/local/kafka/kafka_2.

Kafka 3.3.1, unable to start services in cygwin

2022-11-20 Thread ravi r
I downloaded kafka_2.13-3.3.1.tgz and am unable to start services from a cygwin bash shell on my desktop using zookeeper. Relevant error $ zookeeper-server-start.sh $KAFKA/config/zookeeper.properties [0.003s][error][logging] Invalid decorator '/cygwin64/usr/local/kafka/kafka_2.13-3.3.1/logs

Re: Kafka Streams possible partitioner bug

2022-11-18 Thread Sophie Blee-Goldman
p the docs and also to explicitly log a warning if any of these are set instead of silently ignoring them or flat out breaking as in this case https://issues.apache.org/jira/browse/KAFKA-14404 https://issues.apache.org/jira/browse/KAFKA-14405 On Fri, Nov 18, 2022 at 4:07 PM Upesh Desai wrote: &

Kafka Streams possible partitioner bug

2022-11-18 Thread Upesh Desai
Hello all, We have been working on implementing a custom partitioner for our producer within a simple stream application, that will partition the records by a member field when sending them to the output topic. By looking at the contract of the partition() method in the Partitioner interface,

Re: Out of order messages when kafka streams application catches up

2022-11-15 Thread Sophie Blee-Goldman
Tomasz you'll need to upgrade the kafka Streams dependency to 3.0 (or above) to get the fix that John mentioned before -- this behavior is known/expected on earlier versions such as 2.8.1 as you are using On Tue, Nov 15, 2022 at 2:21 AM Tomasz Gac wrote: > Hi John, > > I've reviewed

Re: Out of order messages when kafka streams application catches up

2022-11-15 Thread Tomasz Gac
Hi John, I've reviewed the test you sent and it seems to be correct, but it may not reproduce our setup. We are using Java 8, kafka-client 2.8.1 with kafka streams version 2.8.1 against the kafka broker version 2.1.1. We are running it as an OSGI bundle with dependencies packaged within

Kafka and Zookeeper FIPS SASL connection

2022-11-14 Thread lk gen
Hi, In my environment I have configured FIPS java, Zookeeper to work with SASL and a Kafka broker to connect to Zookeeper with SASL I noticed that if FIPS is enabled, kafka cannot connect and there is an error "ERROR SASL authentication with Zookeeper Quorum member f

Kafka Streams TLS Behavior

2022-11-08 Thread Rabey, Matthew
Hello, I’ve been managing my team’s Kafka usage for the past few months. Everything is going fine, but I’ve noticed some strange behavior and would like some clarification. Some of our services use the Kafka Streams API in order to communicate with the cluster. When looking at the logs

Re: Kafka broker stuck in a loop

2022-11-08 Thread Nag Pavan Chilakam
wrote: > Hi, > > When we restarted kafka brokers, one of the broker seemed to have got stuck > in a loop. Here is what I see in the log file, that kept repeated > endlessly. > > Appreciate any help you can provide. > > > Oct 29 16:53:54 ip-172- kafka-server-start[244

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-03 Thread Bruno Cadonna
Thank you all! I am really honoured to have become a PMC member. Looking forward to contributing further to Apache Kafka. Best, Bruno On 02.11.22 17:02, Levani Kokhreidze wrote: Congrats Bruno, well deserved! On Wed, 2 Nov 2022, 11:11 Jorge Esteban Quilcate Otoya, < quilcate.jo...@gmail.

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-02 Thread José Armando García Sancio
;>>> > >>>> Congrats! > >>>> > >>>> On Tue, Nov 1, 2022, 15:44 Bill Bejeck > >> wrote: > >>>> > >>>>> Congrats Bruno! Well deserved. > >>>>> > >>>>> -Bill > >>>>> >

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-02 Thread Matthias J. Sax
wrote: Congrats! On Tue, Nov 1, 2022, 15:44 Bill Bejeck wrote: Congrats Bruno! Well deserved. -Bill On Tue, Nov 1, 2022 at 3:36 PM Guozhang Wang wrote: Hi everyone, I'd like to introduce our new Kafka PMC member, Bruno. Bruno has been a committer since April. 2021 and has been very

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-01 Thread Luke Chen
> wrote: > >> > >> Congrats! > >> > >> On Tue, Nov 1, 2022, 15:44 Bill Bejeck > wrote: > >> > >> > Congrats Bruno! Well deserved. > >> > > >> > -Bill > >> > > >> > On Tue, Nov 1, 20

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-01 Thread John Roesler
; Congrats Bruno! Well deserved. >> > >> > -Bill >> > >> > On Tue, Nov 1, 2022 at 3:36 PM Guozhang Wang wrote: >> > >> > > Hi everyone, >> > > >> > > I'd like to introduce our new Kafka PMC member, Bruno. >> > >

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-01 Thread Lucas Brutschy
: > > > > > Hi everyone, > > > > > > I'd like to introduce our new Kafka PMC member, Bruno. > > > > > > Bruno has been a committer since April. 2021 and has been very active in > > > the community. He's a key contributor to Kafka Stre

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-01 Thread Chris Egerton
Congrats! On Tue, Nov 1, 2022, 15:44 Bill Bejeck wrote: > Congrats Bruno! Well deserved. > > -Bill > > On Tue, Nov 1, 2022 at 3:36 PM Guozhang Wang wrote: > > > Hi everyone, > > > > I'd like to introduce our new Kafka PMC member, Bruno. > > > &

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-01 Thread Bill Bejeck
Congrats Bruno! Well deserved. -Bill On Tue, Nov 1, 2022 at 3:36 PM Guozhang Wang wrote: > Hi everyone, > > I'd like to introduce our new Kafka PMC member, Bruno. > > Bruno has been a committer since April. 2021 and has been very active in > the community. He's a key co

[ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

2022-11-01 Thread Guozhang Wang
Hi everyone, I'd like to introduce our new Kafka PMC member, Bruno. Bruno has been a committer since April. 2021 and has been very active in the community. He's a key contributor to Kafka Streams, and also helped review a lot of horizontal improvements such as Mockito. It is my pleasure

Kafka broker stuck in a loop

2022-10-29 Thread Naveen Ede
Hi, When we restarted kafka brokers, one of the broker seemed to have got stuck in a loop. Here is what I see in the log file, that kept repeated endlessly. Appreciate any help you can provide. Oct 29 16:53:54 ip-172- kafka-server-start[2447]: [2022-10-29 16:53:54,654] INFO [Log partition

Re: Granting permission for Kafka Contributor

2022-10-28 Thread Chris Egerton
Hi, You should be good to go now. Cheers, Chris On Fri, Oct 28, 2022 at 4:14 PM yuachieve1234 <260738...@qq.com.invalid> wrote: > Jira ID:yuxj109 > > > > > yuachieve1234 > 260738...@qq.com > > > >

RE: Supported Kafka/Zookeeper Version with ELK 8.4.3

2022-10-28 Thread Kumar, Sudip
Hi Team, We are still waiting for the reply. Please update we must know what version of Kafka is compatible with ELK 8.4 version. Still, I can see no one replied on user and Dev community portal [cid:image001.png@01D8EAFE.9B1F9280] Thanks Sudip From: Kumar, Sudip Sent: Monday, October 17

RE: Kafka Streams - Producer attempted to produce with an old epoch.

2022-10-27 Thread Andrew Muraco
Subject: Re: Kafka Streams - Producer attempted to produce with an old epoch. CAUTION: This email originated from outside of ShopHQ. Do not click links or open attachments unless you recognize the sender and know the content is safe! I'm not one of the real experts on the Producer and even

Re: Kafka Streams - Producer attempted to produce with an old epoch.

2022-10-27 Thread Sophie Blee-Goldman
). How many StreamThreads did you give the app btw? On Thu, Oct 27, 2022 at 8:01 PM Andrew Muraco wrote: > Hi, > I have a kafka streams application deployed on 5 nodes and with full > traffic I am getting the error message: > > org.apache.kafka.common.errors.InvalidProducerEpochExce

<    2   3   4   5   6   7   8   9   10   11   >