Kafka Streams Application Failing to Start Due to State Store Recovery Time Exceeding Producer Transaction Timeout

I have a Kafka Streams application which is currently failing to start due to the following ProducerFencedException: "Caused by: org.apache.kafka.common.errors.ProducerFencedException: task [0_57] Abort sending since producer got fenced with a previous record (key ABCD value [B@4debf146 timesta

flatMapValues does not calculate timestamp for each record generated

Hi, I found out that doing windowed aggregation on records generated by flatMapValues gets incorrect result. Take this topology as an example: myStream.flatMapValues(...) .groupByKey(...) .aggregate(...) Since inside KStreamWindowAggregateProcessor, the timestamp

Re: [VOTE] 1.1.1 RC3

+1 (non-binding) - Ran testAll/releaseTarGzAll on 1.1.0-rc0 tag with jdk-8. - Ran through quickstart of core/streams on builds. - Ran few internal apps targeting to topics on 3 node cluster. Thanks, Satish. On Mon, Jul 9, 2018 at 11:39 PM, Harsha wrote: > +1. > > * Ran unit tests > * Installe

Re: How to create scheduled tasks in Kafka Stream?

Hello Bin, Could you share a bit more on your scenario, i.e. why you'd want this delayed processing functionality so that I can try to think if there's better way to accommodate it? Guozhang On Mon, Jul 9, 2018 at 12:28 PM, Bin Zhu wrote: > Hello Guozhang, > > Thanks for the information. I r

Re: Spark structured streaming + kafka

You were right. It worked. Thanks Pulkit On Mon, Jul 9, 2018 at 5:29 PM, Liam Clarke wrote: > In your SBT script you've specified that your Kafka data source jar is > provided. > > Please read the documentation of dependency scopes. > https://maven.apache.org/guides/introduction/introduction-to

Re: Spark structured streaming + kafka

In your SBT script you've specified that your Kafka data source jar is provided. Please read the documentation of dependency scopes. https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope Kind regards, Liam Clarke On Tue, 10 Jul. 2018, 9:24 am Pu

Spark structured streaming + kafka

Hi, I am trying to do structured streaming with kafka as source. I am unable to get pass of this code. val df = spark .readStream .format("org.apache.spark.sql.kafka010.KafkaSourceProvider") .option("kafka.bootstrap.servers", "localhost:8082") .option("subscribe", "jsontest") .load() T

Re: How to create scheduled tasks in Kafka Stream?

Hello Guozhang, Thanks for the information. I read the document, but that doesn't seem to solve my problem. In my use case, the timers need to be triggered by events and the processing of a event is postponed for a certain amount of time (e.g. 5 minutes). So it's going to be one timer per event.

Re: Real time streaming as a microservice

Hi, I have now successfully created a docker for RHEL75 as follows: [root@rhes75 ~]# docker ps -a CONTAINER IDIMAGE COMMAND CREATED STATUS PORTS NAMES 816f07de15b1zookeeper "/docker-entrypoint.…" 2 hours a

Re: [VOTE] 1.1.1 RC3

+1. * Ran unit tests * Installed in a cluster and ran simple tests Thanks, Harsha On Mon, Jul 9th, 2018 at 6:38 AM, Ted Yu wrote: > > > > +1 > > Ran test suite. > > Checked signatures. > > > > On Sun, Jul 8, 2018 at 3:36 PM Dong Lin < lindon...@gmail.com > wrote: > > > Hello Kafka use

Re: How to create scheduled tasks in Kafka Stream?

Hello Bin, You can take a look at the scheduled punctuate function, which should suit your needs: https://kafka.apache.org/documentation/streams/developer-guide/processor-api.html#defining-a-stream-processor Guozhang On Mon, Jul 9, 2018 at 9:09 AM, Bin Zhu wrote: > Hello, > > Is there any wa

Re: RecordTooLargeException

​you can configure max.request.size ​ when you create Kafka producer, about the second questions, Where is the key size + value size + header size calculated? you can check ` org.apache.kafka.common.record.AbstractRecords.estimateSizeInBytesUp

How to create scheduled tasks in Kafka Stream?

Hello, Is there any way to run scheduled tasks in Kafka Stream? The use case is like: once the event is received, after 5 minutes (no new events are coming at that time), run the job with that event. Is it doable via processor API or DSL? Thanks, Bin

EndToEndLatency failing

Hello my HW/SW is 3 Ubuntu VM with 3 kafka 0.10 ( kerberized) i am passing End2End latency test on my cluster and sometimes i got Exception : "The message read MyTestMessage did not match the message sent "or "only one result expected during this test" There are transient errors ; may i h

RE: Monitoring Kafka

Great ! Thank a lot Daniel ! I will try it. Best Regards Adrien De : Daniel Hanley Envoyé : lundi 9 juillet 2018 16:23:36 À : users@kafka.apache.org Objet : Re: Monitoring Kafka Hi Adrien You could take some ideas from: https://github.com/framiere/monitoring-de

Register now for ApacheCon and save $250

Greetings, Apache software enthusiasts! (You’re getting this because you’re on one or more dev@ or users@ lists for some Apache Software Foundation project.) ApacheCon North America, in Montreal, is now just 80 days away, and early bird prices end in just two weeks - on July 21. Prices will b

Re: Monitoring Kafka

Hi Adrien You could take some ideas from: https://github.com/framiere/monitoring-demo Alternatively, Confluent provide a very powerful Control Center for monitoring and managing Kafka (disclaimer, I work for Confluent!) Best Regards Dan On Mon, Jul 9, 2018 at 2:12 AM, Adrien Ruffie wrote: >

Re: [VOTE] 1.1.1 RC3

+1 Ran test suite. Checked signatures. On Sun, Jul 8, 2018 at 3:36 PM Dong Lin wrote: > Hello Kafka users, developers and client-developers, > > This is the fourth candidate for release of Apache Kafka 1.1.1. > > Apache Kafka 1.1.1 is a bug-fix release for the 1.1 branch that was first > relea

Re: Kafka performance tests

Hi, Thanks a lot for the help, I have found a couple of studies too: https://events.static.linuxfound.org/sites/events/files/slides/HTKafka2.pdf https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing https://cwiki.apache.org/confluence/display/KAFKA/An+analysis+of+the+impact+of+max.i

Re: Very long consumer rebalances

Please re-read the javadoc of KafkaConsumer, make sure you know how to wakeup/close consumer properly while shutting down your application. Try to understand the motivation of KIP-62 and adjust related timeout. On Mon, Jul 9, 2018, 8:05 PM harish lohar wrote: > Try reducing below timer > metada

Ksql

Has anybody used ksql? What are limitations of this and what it’s best for. Would appreciate if you can share some information on this. Best Regards, Praveen Joshi Sent from my iPhone

Re: Very long consumer rebalances

Try reducing below timer metadata.max.age.ms = 30 On Fri, Jul 6, 2018 at 5:55 AM Shantanu Deshmukh wrote: > Hello everyone, > > We are running a 3 broker Kafka 0.10.0.1 cluster. We have a java app which > spawns many consumer threads consuming from different topics. For every > topic we hav

Re: Very long consumer rebalances

Kind people on this group, please help me! On Fri, Jul 6, 2018 at 3:24 PM Shantanu Deshmukh wrote: > Hello everyone, > > We are running a 3 broker Kafka 0.10.0.1 cluster. We have a java app which > spawns many consumer threads consuming from different topics. For every > topic we have specified

Good practice when using kafka with jpa

Dear Kafka community, I would like to draw your attention to the following question: https://stackoverflow.com/questions/51052406/good-practice-when-using-kafka-with-jpa Regards Zbigniew

Monitoring Kafka

Hello Kafka Users, I want to monitor our Kafka cluster correctly. I have read several articles on "how to monitor Kafka" but I have the impression that every company is doing a bit of a thing (rearranging them in his own way). What the really thing I need to monitor, verify and set notifications s

Re: Kafka Consumer Hung on Certain Partitions - Single Kafka Consumer

The processing does not take much time since the application reads from the kafka and immediately puts into a in memory queue and another thread persist these records into DB. On Mon, Jul 9, 2018 at 12:44 PM, dev loper wrote: > Hi Steve, > > When the issue occurs I could see the lag shown for t

Re: Kafka Consumer Hung on Certain Partitions - Single Kafka Consumer

Hi Steve, When the issue occurs I could see the lag shown for the corresponding partitions in the output of kafka-consumer-groups command . I will try changing the log level to debug for kafka consumer and I will post if anything seems to be outside the normal range Thank you Dev Loper On Mon, J