Re: Automation Script : Kafka topic creation

https://github.com/kafka-ops/julie Note that I didn't try it myself, only heard about it... and I am not related to this project בתאריך שבת, 6 בנוב׳ 2021, 12:35, מאת Kafka Life ‏: > Dear Kafka experts > > does anyone have ready /automated script to create /delete /alter topics in > different

Re: Help needed to migrate from one infra to another without downtime

One thing that comes to my mind after reading your explanation, zk quorum should be odd number, you stated you have six zookeepers... I would suggest checking this matter, 3 , 5 , 7 etc... בתאריך יום ד׳, 20 באוק׳ 2021, 22:00, מאת Rijo Roy ‏: > Hi, > > Hope you are safe and well! > > Let me give

Bye bye ZooKeeper ?

I am very curious about what the community apache kafka have in mind for replacement of zookeeper acls , I failed to find any discussion on that... So for Confluent there are MDS and Centeral ACL or RBAC ... What solutions would be there for the open source users, other then staying with

Re: kafka python client to create kafka user/acl info

https://github.com/dpkp/kafka-python/blob/f19e4238fb47ae2619f18731f0e0e9a3762cfa11/test/test_admin_integration.py#L40 בתאריך יום ג׳, 27 ביולי 2021, 16:25, מאת Calvin Chen ‏: > Hi all > > I know there is confluent-kafka-python AdminClient to create/delete > topic/config, can AdminClient also be

Re: Is exactly-once possible with kafka consumer-producer ?

Another acceptable solution is doing idempotent actions while if you re read the message again you will check "did I process it already?" Or doing upsert... and keep it in at least once semantics בתאריך יום ו׳, 16 ביולי 2021, 19:10, מאת Ran Lupovich ‏< ranlupov...@gmail.com>:

Re: Is exactly-once possible with kafka consumer-producer ?

You need to do atomic actions with processing and saving the partition/offsets , while rebalance or assign or on initial start events you read the offset from the outside store, there are documentation and examples on the internet, what type of processing are you doing ? בתאריך יום ו׳, 16 ביולי

Re: Kafka Consumer Retries Failing

I would suggest you will check you bootstrap definition and server.properties, somehow it looks for http://ip:9092 , kafka is not using http protocol, seems something not configured correctly בתאריך יום ג׳, 13 ביולי 2021, 21:46, מאת Rahul Patwari ‏< rahulpatwari8...@gmail.com>: > Hi, > > We are

Re: Kafka ACL: topic with wildcard (*)

As you can see, using only *, it will create some trash  authorizations which you might want to delete בתאריך יום ג׳, 13 ביולי 2021, 16:16, מאת Ran Lupovich ‏< ranlupov...@gmail.com>: > Hello  in order to grant READ access to all topics like you did instead > of just * , which

Re: Kafka ACL: topic with wildcard (*)

Hello  in order to grant READ access to all topics like you did instead of just * , which in this case will take the files in current directory and as the name of the topic, use "*" with the " before and after.. good luck בתאריך יום ג׳, 13 ביולי 2021, 15:40, מאת Dhirendra Singh ‏<

Re: Kafka ACL: topic with wildcard (*)

Use "*" בתאריך יום ג׳, 13 ביולי 2021, 15:40, מאת Dhirendra Singh ‏< dhirendr...@gmail.com>: > Hi All, > I am trying to add an acl for a user to have read permission on all topics. > I executed the following command. > > kafka-acls.sh --bootstrap-server kafka-dev:9092 --command-config >

Re: kafka partition_assignment_strategy

that's sorts things out בתאריך יום ה׳, 1 ביולי 2021, 18:24, מאת Marcus Schäfer ‏< marcus.schae...@gmail.com>: > Hi, > > > Hi Marcus, > > What Ran meant is that please run the following command to check the > > result. > > > > bin/kafka-topics.sh --describe --topic TOPIC_NAME --bootstrap-server

Re: kafka partition_assignment_strategy

I meant kafka-topics describe command , I want to make sure your topic really have 20 partitions בתאריך יום ה׳, 1 ביולי 2021, 14:12, מאת Marcus Schäfer ‏< marcus.schae...@gmail.com>: > Hi, > > > How many partitions do you have for the topic? Can you issue --describe > > please and share, thanks

Re: kafka partition_assignment_strategy

How many partitions do you have for the topic? Can you issue --describe please and share, thanks בתאריך יום ה׳, 1 ביולי 2021, 13:31, מאת Marcus Schäfer ‏< marcus.schae...@gmail.com>: > Hi, > > > Your first understanding is correct, provided each “consumer” means a > > “consumer thread” > > all

Re: Hello, I wonder release date of 2.7.1 kafka.

(and yes, it was released on May 10, 2021) בתאריך יום ג׳, 29 ביוני 2021, 17:49, מאת Ran Lupovich ‏< ranlupov...@gmail.com>: > Kafka 2.7.1 fixes 45 issues since the 2.7.0 release. For more information, > please read the detailed Release Notes > <https://downloads.apac

Re: Hello, I wonder release date of 2.7.1 kafka.

Kafka 2.7.1 fixes 45 issues since the 2.7.0 release. For more information, please read the detailed Release Notes . https://github.com/apache/kafka/releases/tag/2.7.1 Kafka development usually releases minors to previous releases Why

Re: Certificate request not coming mtls

You open seperate topics with no reason... I feel you are spamming the mailing list your need to share more information to be able to investigate and help you... is your server.properties defines the SSL enabled in your port you are trying to connect? You need to setup step by step according

Re: Mtls not working

Can you share your listeners propeties from server.properties בתאריך יום ה׳, 24 ביוני 2021, 19:49, מאת Anjali Sharma ‏< sharma.anjali.2...@gmail.com>: > But in the pcap I am able to see that it is taking some junk certificates > from client side > > On Thu, Jun 24, 2021,

Re: Mtls not working

Make sure that the date and time on the server is correct (The wrong time will cause the SSL certificate connection to fail). בתאריך יום ה׳, 24 ביוני 2021, 19:18, מאת Anjali Sharma ‏< sharma.anjali.2...@gmail.com>: > openssl s_client -connect 10.54.65.99:28105 > socket: Bad file descriptor >

Re: Consumer Group Stuck on "Completing-Rebalance" State

https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-12890 Check out this jira ticket בתאריך יום ב׳, 21 ביוני 2021, 22:15, מאת Tao Huang ‏< sandy.huang...@gmail.com>: > Hi There, > > I am experiencing intermittent issue when consumer group stuck on > "Completing-Reblalance" state.

Re: How to avoid storing password in clear text in server.properties file

Using Confluent Platform you can use feature called Secrets , I am not familiar with open source solution for this. https://docs.confluent.io/platform/current/security/secrets.html בתאריך יום ב׳, 21 ביוני 2021, 16:26, מאת Dhirendra Singh ‏< dhirendr...@gmail.com>: > Hi All, > I am currently

Re: Kafka Ate My Data!

Having setting as described above will tolerate one broker down without service outage, בתאריך יום ו׳, 18 ביוני 2021, 00:42, מאת Ran Lupovich ‏< ranlupov...@gmail.com>: > That's why you have 3 brokers in minimum for production, having > replication factor set to 3 , min.isr set

Re: Kafka Ate My Data!

That's why you have 3 brokers in minimum for production, having replication factor set to 3 , min.isr set to 2, having each broker on different rack , you could also use mm2 or replicator to copy data to other dc... בתאריך יום ו׳, 18 ביוני 2021, 00:33, מאת Jhanssen Fávaro ‏<

Re: kafka 2 way ssl authentication

Share your new configs and logs בתאריך יום ו׳, 4 ביוני 2021, 12:06, מאת Dhirendra Singh ‏< dhirendr...@gmail.com>: > I tried the keytool command suggested by you. still getting the same error. > > On Fri, Jun 4, 2021 at 10:50 AM Ran Lupovich > wrote: > > >

Re: Kafka 2 way authentication not working

lgorithm > ssl.secure.random.implementation > *ssl.client.auth* > > On Fri, Jun 4, 2021, 13:45 Ran Lupovich wrote: > > > All the security configs can be dynamically configured for new listeners. > > In the initial implementation, only some configs will be dynamically

Re: Kafka 2 way authentication not working

https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=74687608#content/view/74687608 Maybe not supported yet? בתאריך יום ו׳, 4 ביוני 2021, 10:49, מאת Ran Lupovich ‏: > Thanks for checking... is there a way for you to check if this behavior is > for "already connec

Re: Kafka 2 way authentication not working

specific nor ssl.client.auth is working dynamically > > On Fri, Jun 4, 2021, 13:04 Ran Lupovich wrote: > > > And not* to specific listener > > > > בתאריך יום ו׳, 4 ביוני 2021, 10:30, מאת Ran Lupovich ‏< > > ranlupov...@gmail.com > > >:

Re: Kafka 2 way authentication not working

And not* to specific listener בתאריך יום ו׳, 4 ביוני 2021, 10:30, מאת Ran Lupovich ‏: > According to documentation it is dynamic and should work, though it is > "general" ssl.auth of the entire broker setting and to specific listener as > you are trying out , but the logic

Re: Kafka 2 way authentication not working

solves the problem but as it is dynamic there is > no need to restart the Kafka right? > > On Fri, Jun 4, 2021, 12:13 Ran Lupovich wrote: > > > Restarting the broker solves the problem? Do your clients fully > disconnect > > and reconnect? > > >

Re: Kafka 2 way authentication not working

but despite doing the same thing it is not working , we are > not getting any client certificate request as such , is there anything that > I am missing in the executing the command or we need to restart the brokers > or anything else we need to do? > > > Thanks & Regards > Anj

Re: Kafka 2 way authentication not working

Adding this information that supports your assumptions that it should be dynamically supportedNotice the update mode - Dynamic Update Mode option in Broker Configurations for the update

Re: kafka 2 way ssl authentication

The default format is jks, use keytool to create a Java KeyStore (JKS) with the certificate and key for use by Kafka. You'll be prompted to create a new password for the resulting file as well as enter the password for the PKCS12 file from the previous step. Hang onto the new JKS password for

Re: In case of Max topic size is reached

criteria with configuration? > > > On Tue, 1 Jun 2021 at 11:36 PM, Ran Lupovich > wrote: > > > They work simultaneously, topic with cleanup policy of DELETE , will > > clean old message older than the retention period and also deletes the > > oldest messages

Re: In case of Max topic size is reached

They work simultaneously, topic with cleanup policy of DELETE , will clean old message older than the retention period and also deletes the oldest messages when retention bytes limit is breached, notice this limit is for each partition in a topic and not total size of the topic, notice as well

Re: Reading offset from one consumer group to use for another consumer group.

יום ו׳, 28 במאי 2021, 21:44, מאת Ran Lupovich ‏: > So I think, You should write to your db the partition and the offset, > while initing the real time consumer you'd read from database where to set > the consumer starting point, kind-of the "exactly once" programming > approa

Re: Reading offset from one consumer group to use for another consumer group.

fen...@gamecircus.com > > > On May 28, 2021, at 12:05 AM, Ran Lupovich > wrote: > > > > > https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seek(org.apache.kafka.common.TopicPartition,%20long) > > > > בתאריך יום ו׳, 28

Re: Issue using Https with elasticsearch source connector

udhari ‏< sunilmchaudhar...@gmail.com>: > Yeah. > I am trying to add truststore in java keystore > Lets see > > On Fri, 28 May 2021 at 5:40 PM, Ran Lupovich > wrote: > > > Anyways you need to remmber it is a java application and you can pass > many > > variable

Re: Issue using Https with elasticsearch source connector

oBalinzo/kafka-connect-elasticsearch-source > > > Thats where I am stuck. > > > Regards, > Sunil. > > On Fri, 28 May 2021 at 9:34 AM, Ran Lupovich > wrote: > > > > > > name=elasticsearch-sinkconnector.class=io.confluent.connect.elasticsearch.Elasticsearc

Re: Reading offset from one consumer group to use for another consumer group.

https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seek(org.apache.kafka.common.TopicPartition,%20long) בתאריך יום ו׳, 28 במאי 2021, 08:04, מאת Ran Lupovich ‏: > While your DB consumer is running you get the access to the partition > ${partition} @

Re: Reading offset from one consumer group to use for another consumer group.

While your DB consumer is running you get the access to the partition ${partition} @ offset ${offset} https://github.com/confluentinc/examples/blob/6.1.1-post/clients/cloud/nodejs/consumer.jswhen setting your second consumers for real time just set them tostart from that point בתאריך יום ו׳, 28

Re: Issue using Https with elasticsearch source connector

/truststore.jkselastic.https.ssl.truststore.password=asdfasdfelastic.https.ssl.truststore.type=JKSelastic.https.ssl.protocol=TLS בתאריך יום ו׳, 28 במאי 2021, 07:03, מאת Ran Lupovich ‏: > https://docs.confluent.io/kafka-connect-elasticsearch/current/security.html > > בתאריך יום ו׳, 28 במאי 2021, 07:00, מאת sunil chaudhari ‏< >

Re: Issue using Https with elasticsearch source connector

> בתאריך יום ה׳, 27 במאי 2021, 19:55, מאת sunil chaudhari ‏< > > sunilmchaudhar...@gmail.com>: > > > > > Hi Ran, > > > That problem is solved already. > > > If you read complete thread and see that last problem is about https > > > connection. > > >

Re: Kafka getting down every week due to log file deletion.

The main purpose of the /*tmp* directory is to temporarily store *files* when installing an OS or software. If any *files* in the /*tmp* directory have not been accessed for a while, they will be automatically *deleted* from the system בתאריך יום ה׳, 27 במאי 2021, 19:04, מאת Ran Lupovich

Re: Issue using Https with elasticsearch source connector

t; Hi Ran, > That problem is solved already. > If you read complete thread and see that last problem is about https > connection. > > > On Thu, 27 May 2021 at 8:01 PM, Ran Lupovich > wrote: > > > Try setting es.port = "9200" without quotes? > > >

Re: Kafka getting down every week due to log file deletion.

Seems you log dir is sending your data to tmp folder, if I am bot mistken this dir automatically removing files from itself, causing the log deletuon procedure of the kafka internal to fail and shutdown broker on file not found בתאריך יום ה׳, 27 במאי 2021, 17:52, מאת Neeraj Gulia ‏<

Re: Issue using Https with elasticsearch source connector

Try setting es.port = "9200" without quotes? בתאריך יום ה׳, 27 במאי 2021, 04:21, מאת sunil chaudhari ‏< sunilmchaudhar...@gmail.com>: > Hello team, > Can anyone help me with this issue? > > > https://github.com/DarioBalinzo/kafka-connect-elasticsearch-source/issues/44 > > > Regards, > Sunil. >

Re: Modify kafka-connect api context path

--server.servlet.context-path="/kafdrop" Something like this ? https://github.com/obsidiandynamics/kafdrop/issues/9 בתאריך יום ד׳, 26 במאי 2021, 23:44, מאת Fernando Moraes ‏< fernandosdemor...@gmail.com>: > Hello, I would like to know if it is possible to modify via config > properties the

Re: Weird behavior of topic retention - some are cleaned up too often, some are not at all

Sorry I did not see all the info at first, what do you mean by topic getting cleaned, you have setting to check retention every 5 minutes, the data that getting "cleaned" is the older data which is 30 days old... am I missing something? בתאריך יום ג׳, 25 במאי 2021, 23:04, מאת Ra

Re: Weird behavior of topic retention - some are cleaned up too often, some are not at all

By the segment size you are "delete" after 1 giga bytes is full , per partition, you need to remmber the retention is done when segments closed , per partition בתאריך יום ג׳, 25 במאי 2021, 22:59, מאת Ran Lupovich ‏: > Have you checked the segment size? Did you decribe the topic &g

Re: Weird behavior of topic retention - some are cleaned up too often, some are not at all

Have you checked the segment size? Did you decribe the topic configuration?maybe you created it with some settings you dont remember בתאריך יום ג׳, 25 במאי 2021, 19:51, מאת Marina Popova ‏: > > Any idea what is wrong here? I have restarted Kafka brokers a few times, > and all other Confluent

Re: kafka command is not working

In bootstrap server put at least two brokers nodes of the cluster to get the metadata of the initial connection בתאריך יום ב׳, 17 במאי 2021, 18:07, מאת Aniket Pant ‏: > Hi team, > my question is , i have 3 nodes of kafka cluster and when i stop one broker > i cannot lag messages it show me error

Re: Kafka SSL

https://docs.confluent.io/platform/current/kafka/authentication_ssl.html Check this out בתאריך יום ו׳, 30 באפר׳ 2021, 20:06, מאת Ran Lupovich ‏< ranlupov...@gmail.com>: > Hi seems you setup in port 9093 only ssl as a method of authentication and > method of transfer encryption,

Re: Kafka SSL

Hi seems you setup in port 9093 only ssl as a method of authentication and method of transfer encryption, so it means in the client configuration you would need the keystore configured as well, you could choose other mean of authentication such as PLAIN_SSL or so own, hope thats helps, keep us

Re: Standard way to get http POST request into a Kafka topic?

Btw. Just now accomplished a working poc in dev using wso2 , confluent rest proxy , confluent schema registry, kafka Produce message to kafka via post http rest request בתאריך יום ד׳, 28 באפר׳ 2021, 06:42, מאת Ran Lupovich ‏< ranlupov...@gmail.com>: > Hi, have a look for Rest Proxy

Re: Standard way to get http POST request into a Kafka topic?

Hi, have a look for Rest Proxy component as part of the kafka eco system בתאריך יום ד׳, 28 באפר׳ 2021, 01:27, מאת Reed Villanueva ‏< villanuevar...@gmail.com>: > What is the best-practice/kafka way to get http(s) POST requests into a > Kafka topic (kafka v2.0.0 installed on a HDP cluster)? >

Re: JDBC Sink Connector: Counterpart to Dead Letter Queue to keep track of successfully processed records

Just a thought , what about jdbc source connector from the database to the succsess topic? בתאריך יום ד׳, 28 באפר׳ 2021, 00:19, מאת Florian McKee ‏< florian.mc...@gmail.com>: > Hi, > > > I want to ingest messages that are sent by 3rd parties into our system. > > > The ingestion process is as

Re: How to emit lag into Prometheus?

Did you test this out https://github.com/lightbend/kafka-lag-exporter ? We are using bash scripting to describe the consumer groups from the cluster side into file and ingest to splunk for dashboarsing... Note - Approach of describing from outside the consumer does not give the 'real time' view

Re: [kafka-clients] Re: [ANNOUNCE] Apache Kafka 2.8.0

Hello, Maybe I missed it in the documentations but where can I read about what is the future plan for the zookeeper managed ACLs ? בתאריך יום ב׳, 19 באפר׳ 2021, 22:48, מאת Israel Ekpo ‏: > This is fantastic news! > > Thanks everyone for contributing and thanks John for managing the release. > >

Re: options for kafka cluster backup?

I guess that in case of avoiding data lose you would need to use 3 replica in different rack/sites awareness to avoid data lose, Confluent's Replicator or MirrorMaker are for copying data from one cluster to another usually in different dc / regions, If I am not mistaken בתאריך יום ו׳, 5 במרץ