Re: Reg: Dual Kafka Clients - Issues with jar version conflicts

Hello Gowtham, do you think it is possible to elaborate on your use case? what are you trying to solve? On Wed, Apr 15, 2020 at 2:47 PM Gowtham S wrote: > Hello, > > It will be helpful if anyone suggests solutions. > > Thanks and regards, > Gowtham S, MCA > > > On Tue, 7 Apr 2020 at 12:31,

Re: [kafka-clients] [VOTE] 2.5.0 RC3

Hello, I have ran the tests (passed) Follow the quick start guide with scala 2.12 (success) +1 Thanks! -- Jonathan On Tue, Apr 14, 2020 at 1:16 AM Colin McCabe wrote: > +1 (binding) > > verified checksums > ran unitTest > ran check > > best, > Colin > > On Tue, Apr 7, 2020, at 21:03, David

Re: callback function

Hello, when you say "something is new in the queue" you mean a new message/record is available in a partition within a topic? Cheers! On Tue, Jan 7, 2020, 8:46 PM Tavares Forby wrote: > Hi, > > Is there a method in which a function can wake when something is new in > the queue? This has to be

Re: Running Kafka on a single node machine

my question is, will it be possible to run > > kafka > > > on a single node/machine as a production. We are using k3s as an > > > orchestration so whenever kafka goes down it can just restart it. > > > > > > Hoping to hear from you soon. Thank you! > > > > > > *Sincerely,* > > > > > > *Matthew Aldrin S. Torres* > > > > > > > > > -- > > Santilli Jonathan > > > -- Santilli Jonathan

Re: Running Kafka on a single node machine

Hello, running Kafka as a single Broker or a cluster of one node voids the whole purpose you are trying to achieve "...robustness of our system by using kafka..." Just think that is that single node goes down, what's gonna happen? Kafka besides the performance is well known by the availability.

Re: Kafka transactions not working in 2.3.1

tocol.version > though. Would reverting back to 0.10.1 and upgrading again with > *log.message.format.version > *work? > > Thanks, > Soman > > On Sun, Dec 8, 2019 at 1:40 PM Jonathan Santilli < > jonathansanti...@gmail.com> > wrote: > > > Hard to tell wit

Re: Kafka transactions not working in 2.3.1

: > Hello Jonathan, > I've installed the same version on all the brokers. The brokers were > restarted as well. Any suggestions on how I can fix this? > > Thanks, > Soman > > On Sat, Dec 7, 2019 at 9:17 AM Jonathan Santilli < > jonathansanti...@gmail.com> > wrote: &

Re: Kafka transactions not working in 2.3.1

Hello, have you ensured you have installed the same version in all brokers? Did you restart all brokers after the update as indicated in the rolling upgrade instructions? Cheers! On Sat, Dec 7, 2019, 2:38 PM Soman Ullah wrote: > Hello, > > I recently upgraded our kafka cluster from Kafka

Re: Re: The question of kafka

c: sqyc_kid_bigdata Partition: 13 Leader: 1 Replicas: 1,2 Isr: 1 > Topic: sqyc_kid_bigdata Partition: 14 Leader: 3 Replicas: 2,3 Isr: 3 > > > > > > > > > At 2019-11-25 23:22:22, "Jonathan Santilli" > wrote: > >Hello, > > > >do you mean why repli

Re: The question of kafka

Hello, do you mean why replicas are not in sync? I think you need to be more specific, please. Cheers! -- Jonathan On Mon, Nov 25, 2019 at 2:54 PM 房延鹏 <13120336...@163.com> wrote: > Hi， > >I want to ask a question. Why replicas not contain the isr. The version > is kafka_2.11-0.11.00.

Re: Transactional Producer

t; non-transactional producer publishing to the same partition? > > Sergi > > вс, 24 нояб. 2019 г. в 21:12, Jonathan Santilli < > jonathansanti...@gmail.com > >: > > > Hello Sergi, > > > > 1. Is it OK to mix transactional and non-transactional ap

Re: Transactional Producer

Hello Sergi, 1. Is it OK to mix transactional and non-transactional approach with a single KafkaProducer instance? - This is not possible, a transactional producer can not send data outside a transaction. I mean sometimes I want to publish multiple messages transactionally, but oftentimes just a

Re: Kafka 2.2.1 with OpenJDK 11

Hello, I can confirm that the version 2.2.0 supports OpenJDK 8 in the broker side, we are using it on production. Cheers! -- Jonathan On Sat, Oct 26, 2019, 4:47 PM Debraj Manna wrote: > Kafka latest doc mention about JDK 8. But make no mention of openjdk or > Oracle JDK. Can someone confirm

Re: [VOTE] 2.3.1 RC2

Hello David, thanks for the release, +1 I have run the tests (all passed) and executed some internal Apps successfully after starting one Broker following the quick start guide. Cheers! -- Jonathan On Tue, Oct 22, 2019 at 11:49 PM Colin McCabe wrote: > +1. I ran the broker, producer,

Re: Kafka BootStrap : Error while deleting the clean shutdown file in dir /tmp/data (kafka.server.LogDirFailureChannel) : Caused by: OOM: Map failed

Hello Senthil, I would recommend to not have the data in /tmp/data. Also as a recommendation, set to equal values the -Xmx and -Xms parameters. Cheers! -- Jonathan On Wed, Sep 4, 2019 at 1:27 PM Karolis Pocius wrote: > I had the same issue which was solved by increasing max_map_count >

Re: Failed to rebalance -- Unexpected error in InitProducerIdResponse

for the quick response. I have, but I didn't see any > errors. > > > > On Tue, Sep 3, 2019 at 1:43 PM Jonathan Santilli < > jonathansanti...@gmail.com> > wrote: > > > Hello, have you check the Brokers Logs? maybe there is something there at > > that

Re: Failed to rebalance -- Unexpected error in InitProducerIdResponse

Hello, have you check the Brokers Logs? maybe there is something there at that particular time when the error occurred in the App side. Cheers! On Tue, Sep 3, 2019 at 12:38 PM Dimitry Lvovsky wrote: > Hi all, > We have a streaming app running for over a year now without much incident > until

Uncaught error in kafka producer I/O thread

Hello, hope you all are well, I would like to know if you guys also have seen this issue in your Apps logs: ERROR [kafka-producer-network-thread | producer-5] internals.Sender (Sender.java:240) - [Producer clientId=producer-5, transactionalId=client-XXX-1-4] Uncaught error in kafka producer I/O

Re: Consumer poll / fetch timeout

Hello, please, share the config of your consumer and any Exception you see in the logs, please. Cheers! On Wed, Aug 21, 2019 at 4:30 PM Shreesha Hebbar wrote: > > > > Hi all, > >Noticing some of the consumer fetch timing out (2-10 sec), even though > > there is a steady rate of

Re: RecordTooLargeException on 16M messages in Kafka?

me know, cheers! -- Jonathan On Thu, Aug 15, 2019 at 8:09 PM l vic wrote: > setting max.request.size worked when i did it in producer code; don't > understand what producer.properties is for, seems like it's not used. > > On Thu, Aug 15, 2019 at 2:25 PM Jonathan Santilli <

Re: RecordTooLargeException on 16M messages in Kafka?

I have asked because I did not see that in your previous email when you tried the console producer. Jonathan. On Thu, Aug 15, 2019, 3:07 PM l vic wrote: > yes, in producer.properties > > On Thu, Aug 15, 2019 at 9:59 AM Jonathan Santilli < > jonathansanti...@gmail.com>

Re: RecordTooLargeException on 16M messages in Kafka?

Hello, try to send and flush just one message of 16777239 bytes, to verify the error still shows up. Cheers! -- Jonathan On Thu, Aug 15, 2019 at 2:23 AM l vic wrote: > My kafka (1.0.0) producer errors out on large (16M) messages. > ERROR Error when sending message to topic test with key:

Re: Doubts in Kafka

Hello Aruna, if the duplication you are referring to is the duplication of the events/records that arrive and are consumed to/from Kafka, exactly-once semantics and transactions are what you are looking for. Kafka is prepared (since version 0.11 IIRC) to support exactly once, it means that events

Re: org.apache.kafka.streams.processor.TimestampExtractor#extract method in version 2.3 always returns -1 as value

rote: > > > Sounds like a regression to me. > > > > We did change some code to track partition time differently. Can you > > open a Jira? > > > > > > -Matthias > > > > On 6/26/19 7:58 AM, Jonathan Santilli wrote: > > > Sure Bill

Re: org.apache.kafka.streams.processor.TimestampExtractor#extract method in version 2.3 always returns -1 as value

produce on my > end? > > -Bill > > > On Wed, Jun 26, 2019 at 10:23 AM Jonathan Santilli < > jonathansanti...@gmail.com> wrote: > > > Hello Bill, > > > > am implementing the TimestampExtractor Interface, then using it to > consume, > > lik

Re: org.apache.kafka.streams.processor.TimestampExtractor#extract method in version 2.3 always returns -1 as value

etting the default.timestamp.extractor config value. Cheers! -- Jonathan On Wed, Jun 26, 2019 at 3:16 PM Bill Bejeck wrote: > Hi Jonathan, > > Thanks for reporting this. Which timestamp extractor are you using in the > configs? > > Thanks, > Bill > > On Wed, Jun 26, 2019 at 9:14 AM Jo

org.apache.kafka.streams.processor.TimestampExtractor#extract method in version 2.3 always returns -1 as value

Hello, hope you all are doing well, am testing the new version 2.3 for Kafka Streams specifically. I have noticed that now, the implementation of the method extract from the interface org.apache.kafka.streams.processor.TimestampExtractor *public* *long* extract(ConsumerRecord record, *long*

Re: Repeating UNKNOWN_PRODUCER_ID errors for Kafka streams applications

Hello, we currently see the same WARN logs when the App is processing low rate of records, as soon the app starts processing a high volume of records, those WARN stop showing in the logs. According to other email threads, this should be fixed with https://issues.apache.org/jira/browse/KAFKA-7190.

Re: Kafka ISR to server count questions

Hello Scott, as you see, the logs are telling you what is happening: The size of the current ISR Set(1) is insufficient to satisfy the min.isr requirement of 2 for partition . It means you either increase the replication factor > 2 or decrease the min.isr to 1. Since currently your replication

Re: [ANNOUNCE] Apache Kafka 2.2.1

That's fantastic! thanks a lot Vahid for managing the release. -- Jonathan On Mon, Jun 3, 2019 at 5:18 PM Mickael Maison wrote: > Thank you Vahid > > On Mon, Jun 3, 2019 at 5:12 PM Wladimir Schmidt > wrote: > > > > Thanks Vahid! > > > > On Mon, Jun 3, 2019, 16:23 Vahid Hashemian wrote: >

Re: Transactional markers are not deleted from log segments when policy is compact

5 AM Pranavi Chandramohan < > prana...@campaignmonitor.com> wrote: > > > Thanks Jonathan! That should help. > > > > On Fri, 31 May 2019, 6:44 pm Jonathan Santilli, < > > jonathansanti...@gmail.com> wrote: > > > >> Hello Pranavi, > >&g

Re: Transactional markers are not deleted from log segments when policy is compact

Hello Pranavi, it sounds like this was solved in the release candidate 2.2.1-RC1 ( https://issues.apache.org/jira/browse/KAFKA-8335) Take a look at it. Hope that helps. Cheers! -- Jonathan On Fri, May 31, 2019 at 8:59 AM Pranavi Chandramohan < prana...@campaignmonitor.com> wrote: > Hi

Re: Transaction support multiple producer instance

Hello Wenxuan, there reason of the Exception, by design the transaction Id must be unique per producer instance, this is from the Java docs: https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html "The purpose of the transactional.id is to enable transaction

Re: Restart of kafka-streams with more than one partition on topic is reprocessing the data from beginning of the topic

gt; > Thanks for the suggestion. It is working fine with the kafka 2.2.1-rc1 > vesrion. > Do we have any tentative release date for this version. Please let me know > if you have any info on this. > Thanks in Advance. > > Thanks > Kalyani > > On 2019/05/24 08:00:33, Jon

Re: quick start does not work...

Hello Tom, It is great that you figure it out. About the ">" is what is used to represent the prompt where the user will type the command. Am not sure is a good idea to remove it, however there is always room for improvement. Cheers, -- Jonathan On Sat, May 25, 2019, 6:51 PM Tom Kwong wrote:

Re: Restart of kafka-streams with more than one partition on topic is reprocessing the data from beginning of the topic

Hello Kalyani, try testing the RC kafka-2.2.1-rc1, for what you describe seems to be a problem that has been solved in the version 2.2.1 ( https://issues.apache.org/jira/browse/KAFKA-7895) (which is under voting right now 2.2.1-RC1

Re: quick start does not work...

Hello Tom, did you follow all the steps as instructed in the quickstart? the log is an INFO, while doing it, you will notice logs with the message "Error:KeeperErrorCode = NoNode" in different places, this is normal, is the way it works or how have been designed the interaction from the Broker to

Re: Kafka delaying message

Maybe you could: 1.- Disable auto.commit on the consumer side. 2.- Consume messages, one by one in each poll. 3.- Check the record timestamp a.- If the timestamp is >= desired window (time slot), process it and commit offset b.- If the timestamp is < desired window (time slot), do not

Re: Stream application :: State transition from PARTITIONS_REVOKED to PARTITIONS_ASSIGNED continuously

Hello Nayanjyoti, about this part you mentioned: "Also, we had noticed that on restarts the downstream of the suppress operator is *flooded* with events, which in the ideal case wouldn't have come. I came across https://stackoverflow.com/a/54227156 where Matthias had responded the behaviour of

Re: [kafka-clients] Re: [VOTE] 2.2.1 RC1

Hello, I have run the tests, all tests passed. I have also run the quick start and test my apps with that commit 55783d313, everything worked. +1 Thanks a lot for the release, I have been waiting for a bug fix on this release for long, yuju! :) Cheers! -- Jonathan On Sat, May 18, 2019 at

Broker does not stop gracefully with SIGTERM

Hello, hope you all are doing well, Am trying to stop gracefully a Broker with SIGTERM (-15). After almost 12 hours the process is still alive. I do not see any data/replication going in or out from this Broker. The following are the logs immediately after sending the SIGTERM signal to the

Re: [VOTE] 2.2.1 RC0

Hello Vahid, am testing one of our Kafka Stream Apps with the 2.2.1-rc, after few minutes, I see this WARN: 2019-05-09 13:14:37,025 WARN [test-app-id-dc27624a-8e02- 4031-963b-7596a8a77097-StreamThread-1] internals.ProcessorStateManager ( ProcessorStateManager.java:349) - task [0_0] Failed to

Re: Restart process after adding control.plane.listener.name config in Kafka 2.2.0

ener name, e.g. "CONTROLLER". During rolling upgrade of the 2nd > round, > > controller to brokers traffic will start using the " > > control.plane.listener.name", and go through the proposed changes in > this > > KIP. > > > On Fri

Restart process after adding control.plane.listener.name config in Kafka 2.2.0

Hello, hope you all are great, I would like to know what's the process to update the Kafka Broker configuration in order to use the new config: *control.plane.listener.name in Kafka version 2.2.0* The documentation says: *Name of listener used for

Re: Slow cluster recover after a restart

help/guide will be highly appreciated. Cheers! -- Jonathan On Thu, Apr 25, 2019 at 3:57 PM Jonathan Santilli < jonathansanti...@gmail.com> wrote: > [image: image.gif] > Hello, > > we are updating one of our clusters from version 2.0 to 2.2. The cluster > has 4 brokers.

Slow cluster recover after a restart

Hello, we are updating one of our clusters from version 2.0 to 2.2. The cluster has 4 brokers. After stopping the first broker the cluster was still operating normally as expected, receiving data from producers and sending data to consumers. When starting the first broker again with the new

Re: Kafka point in time recovery?

Hello Kumar, if I got it correctly, you can try with the method offsetsForTimes from the KafkaConsumer Api. Maybe you can reset the offset to a particular timestamp and consume up to a certain offset or timestamp.

Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax

Definitely well deserved! Congrats Matthias! On Thu, Apr 18, 2019, 10:46 PM Bill Bejeck wrote: > Congrats Matthias! Well deserved! > > -Bill > > On Thu, Apr 18, 2019 at 5:35 PM Guozhang Wang wrote: > > > Hello Everyone, > > > > I'm glad to announce that Matthias J. Sax is now a member of Kafka

Re: __consumer_offsets topic, some partitions with the last offset equals to zero

ve less than 50 group/topic/partition combination in your cluster, > it could make sense to have __consumer_offsets partitions without any > message > > Best regards > > On Tue, Apr 9, 2019 at 12:43 PM Jonathan Santilli < > jonathansanti...@gmail.com> wrote: > > &

__consumer_offsets topic, some partitions with the last offset equals to zero

Hello, hope you are doing well, We have noticed that the last offset for some of the partitions for topic *__consumer_offsets is zero*. For instance, the following is a testing cluster running version 2.2, this cluster has been running for about 15 months already: *./bin/kafka-run-class.sh

Re: KTable.suppress(Suppressed.untilWindowCloses) does not suppress some non-final results when the kafka streams process is restarted

ow. > > > > I don't think it's related to the duplicates, but that shutdown error is > > puzzling. Can you print the topology (with topology.describe() ) ? This > > will tell us what is in task 1 (i.e., *1_*) of your program. > > > > Thanks, > > -John >

Re: [VOTE] 2.2.0 RC1

Hello Matthias, I have run the tests without issue. Also, I have performed the quick start successfully, thanks for the release. Cheers! -- Jonathan On Wed, Mar 6, 2019 at 8:58 PM Matthias J. Sax wrote: > Thanks for testing the RC Jakub! > > I agree, that both issue are no reason to cancel

Re: KTable.suppress(Suppressed.untilWindowCloses) does not suppress some non-final results when the kafka streams process is restarted

! -- Jonathan On Fri, Mar 1, 2019 at 5:11 PM Jonathan Santilli wrote: > Hello John, hope you are well. > I have tested the version 2.2 release candidate (although I know it has > been postponed). > I have been following this email thread because I think am experiencing > the sam

Re: Alternative of (kafka-consumer-groups.sh --describe) in java clients library to get consumer lag

Maybe you can take a look to https://github.com/yahoo/kafka-manager It allows you to get the info in a Webpage though. Cheers! -- Jonathan On Fri, Mar 1, 2019 at 1:19 PM Arunim Chopra wrote: > Hi Randall, > There are many monitoring tools which gives the exact information as >

Re: KTable.suppress(Suppressed.untilWindowCloses) does not suppress some non-final results when the kafka streams process is restarted

Hello John, hope you are well. I have tested the version 2.2 release candidate (although I know it has been postponed). I have been following this email thread because I think am experiencing the same issue. I have reported in an email to this list and also all the details are in OS (

Re: [kafka-clients] [VOTE] 2.1.1 RC2

Hello, I have downloaded the source and executed integration and unit tests successfully. Ran kafka-monitor for about 1 hour without any issues. +1 Thanks for the release Colin. -- Jonathan Santilli On Fri, Feb 15, 2019 at 6:16 AM Jason Gustafson wrote: > Ran the quickstart agai

Re: Caused by: java.lang.IllegalArgumentException: Out of order offsets found in List

Hello Rohit, maybe is better to fill a JIRA with some context, details and how to reproduce it? like this is kind of difficult to detect the root cause unless another person in the list actually can provide some insights with the provided information. Cheers! -- Jonathan On Tue, Feb 12, 2019

Re: [VOTE] 2.1.1 RC1

-2.1.1.tgz and ran a cluster of 3 Brokers. Kept Kafka-monitor running for about 1 hour without any issues. +1 Thanks for the release Colin. -- Jonathan Santilli On Wed, Jan 30, 2019 at 8:18 PM Eno Thereska wrote: > I couldn't repro locally, that was on an m3.large. And it's not happen

Re: Urgent : Kafka-Client Download installation

Hello Kumar, you can either build your own .jar client from the source or download from a repository like https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients/2.1.0 and use the artefact you need (mvn, SBT, gradle, ivy...) The

Re: Why do the offsets of the consumer-group (app-id) of my Kafka Streams Application get reset after application restart?

> can confirm how much data and from which offset the streams is restoring a > state store by using a custom implementation of the StateRestoreListener > class and set it via the KafkaStreams#setGlobalStateRestoreListener. > > -Bill > > > On Mon, Jan 14, 2019 at 7:32 AM Jonat

Why do the offsets of the consumer-group (app-id) of my Kafka Streams Application get reset after application restart?

I have a Kafka Streams application for which, whenever I restart it, the offsets for the topic partitions (*KTABLE-SUPPRESS-STATE-STORE*) it is consuming get reset to 0. Hence, for all partitions, the lags increase and the app needs to reprocess all the data. I have ensured the lag is 1 for every

Re: Maximum limit of partition in Kafka topic

I would recommend taking a look at this presentation: https://www.slideshare.net/ConfluentInc/how-much-kafka/ConfluentInc/how-much-kafka At the end of the presentation, there are also some useful links. Cheers! -- Jonathan On Tue, Jan 8, 2019 at 2:01 PM marimuthu eee wrote: > Hi, > > How

Re: Dynamic Partitioning

Hello Marimuthu, Partitions are created at the moment the topic is created, however, you can also add more new partitions to a topic once have been created. There is not a possibility to create partitions dynamically (out-of-the-box) into a topic unless you create your own business logic and/or

Re: One partitions segment files in different log-dirs

Hello Margus, Am not sure if I got your question correctly, but, assuming you have a topic called "*kafka-log*" with two partitions, each of them (kafka-log-1 and kafka-log-2) will contain its own segments. Kafka Brokers will distribute/replicate (according to the Brokers config) the topics

DisconnectException stop Kafka Stream app consuming from topic

Hello, hope you all are great today! am using a Kafka Stream application: ... *final* StreamsBuilder streamsBuilder = *new* StreamsBuilder(); *final* KStream, MyObject> myObjects = streamsBuilder .stream(inputTopicNames, Consumed.*with*( myObjectsWindowSerde,

Re: Setup GC log rotate policy for kafka broker servers

Hello Parth, I would recommend avoiding rapid restarts, it means, restarting the Brokers immediately one after the other, also (in my opinion, avoid restarting the Brokers all at once). According to my experience and my opinion, it is better to restart one by one, but wait until the restarted

Re: [VOTE] 2.1.0 RC1

Hello, +1 I have downloaded the release artifact from http://home.apache.org/~lindong/kafka-2.1.0-rc1/ Executed a 3 brokers cluster. (java8 8u192b12) Executed kafka-monitor for about 1 hour without problems. Thanks, -- Jonathan On Fri, Nov 9, 2018 at 11:33 PM Dong Lin wrote: > Hello Kafka

Re: Deciding number of partitions

Hello Sachit, I think that's the first question all of us ask our self when facing that moment, "Ok, but, how many partitions do we need?" I would say, according to my experience, that directly depends on your business and the kind of problem you are trying to solve, how many consumer and

Re: Visualizing kafka topics data

Hello Akash, we use for simple cases Kafdrop: https://github.com/HomeAdvisor/Kafdrop Is a jar that raises an HTTP server. Hope it helps. -- Jonathan On Sat, Aug 4, 2018 at 12:56 AM Rahul Singh wrote: > Graylog, or Kibana (connected to ElasticSearch) > > > Rahul > On Aug 3, 2018, 5:12 PM

Re: Error while creating ephemeral at /brokers/ids/BROKER_ID

Thanks a lot for your reply Ismael, I have filled the JIRA: https://issues.apache.org/jira/browse/KAFKA-7165 Cheers! -- Jonathan On Sat, Jul 14, 2018 at 5:25 PM Ismael Juma wrote: > Hi Jonathan, > > Can you please file a JIRA? > > Ismael > > On Sat, Jul 14, 2018 at 3:44

Error while creating ephemeral at /brokers/ids/BROKER_ID

Hello, hope you all are Ok, I would like to know if this is the expected behavior: ERROR Error while creating ephemeral at /brokers/ids/BROKER_ID, node already exists and owner '*216186131422332301*' does not match current session '*288330817911521280*' (kafka.zk.KafkaZkClient$CheckedEphemeral)