Can two different kafka clusters be used in kafka streams for consumption from 1 cluster and produce on another cluster

Hi All, We are using a streams application and currently the application uses a common kafka cluster that is shared along with many other applications. Our application consumes from topics that are populated by other applications and it consumes the events from those topics, processes those

Re: kafka streams consumer group reporting lag even on source topics removed from topology

cs), and > thus your app would start with newly created (and thus empty topics). -- > You might want to restart the app with `auto.offset.reset = "earliest"` > and reprocess all available input to re-create state. > > > -Matthias > > On 8/19/23 8:07 AM, Pushkar D

Re: kafka streams consumer group reporting lag even on source topics removed from topology

pic basis, too. > > > HTH, >-Matthias > > > On 8/16/23 2:11 AM, Pushkar Deole wrote: > > Hi streams Dev community @matthias, @bruno > > > > Any inputs on above issue? Is this a bug in the streams library wherein > the > > input topic removed from st

Re: kafka streams consumer group reporting lag even on source topics removed from topology

Hi streams Dev community @matthias, @bruno Any inputs on above issue? Is this a bug in the streams library wherein the input topic removed from streams processor topology, the underlying consumer group still reporting lag against those? On Wed, Aug 9, 2023 at 4:38 PM Pushkar Deole wrote: >

kafka streams consumer group reporting lag even on source topics removed from topology

Hi All, I have a streams application with 3 instances with application-id set to applicationV1. The application uses processor API with reading from source topics, processing the data and writing to destination topic. Currently it consumes from 6 source topics however we don't need to process

Re: kafka streams re-partitioning on incoming events

an > other processor you can then read the repartitioned events from that > topic. However, you have to manage the intermediate topic yourself. > > In the DSL there is the process() method that allows to use custom code > for processing events similar to the Processor API. > &g

Re: kafka streams re-partitioning on incoming events

ce > partitions. > > Best, > Bruno > > On 7/9/23 2:45 PM, Pushkar Deole wrote: > > Hi, > > > > We have a kafka streams application that consumes from multiple topic > with > > different keys. Before processing these events in the application, we >

Re: kafka streams re-partitioning on incoming events

Hello, *Kafka dev community, @matthiasJsax* Can you comment on below question? It is very important for us since we are getting inconsistencies due to current design On Sun, Jul 9, 2023 at 6:15 PM Pushkar Deole wrote: > Hi, > > We have a kafka streams application that consumes from

kafka streams re-partitioning on incoming events

Hi, We have a kafka streams application that consumes from multiple topic with different keys. Before processing these events in the application, we want to repartition those events on a single key that will ensure related events are processed by same application instance. e.g. the events on

Re: kafka streams partition assignor strategy for version 2.5.1 - does it use sticky assignment

u can > adjust to speed up the warm-up process: > https://cwiki-test.apache.org/confluence/display/KAFKA/KIP-441%3A+Smooth+Scaling+Out+for+Kafka+Streams#KIP441:SmoothScalingOutforKafkaStreams-Parameters > . > > I hope this helps! > -John > > On 2023/04/14 17:41:19 Pushkar D

Re: kafka streams partition assignor strategy for version 2.5.1 - does it use sticky assignment

Any inputs on below query? On Wed, Apr 12, 2023 at 2:22 PM Pushkar Deole wrote: > Hi All, > > We are using version 2.5.1 of kafka-streams with 3 application instances > deployed as 3 kubernetes pods. > It consumes from multiple topics, each with 6 partitions. > I would like

kafka streams partition assignor strategy for version 2.5.1 - does it use sticky assignment

Hi All, We are using version 2.5.1 of kafka-streams with 3 application instances deployed as 3 kubernetes pods. It consumes from multiple topics, each with 6 partitions. I would like to know if streams uses sticky partition assignor strategy internally since we can't set it externally on streams.

is exactly-once supported with kafka streams application with external state store like redis

Hi All, We are using streams application with redis for state store. Redis was mainly considered instead of kafka state stores because of the reason that global state store once updated by one application instance was taking few milliseconds to reflect updated global state to another application

can Kafka streams support ordering across 2 different topics when consuming from multiple source topics?

Hi All, We have a kafka streams application that consumes from 2 different topics say topic A and topic B. The application uses data of telephone call on those topics and each call has a call id which is used as key to send events to those 2 topics. e.g. for a telephone call, the 1st event

Re: same keys appearing in state stores on different pods when using branches in kafka streams

the topology description? That will show us > whether we should expect the stores to always be in the same instances or > not. If you can also include a simplified version of your program, we might > be able to provide some suggestions. > > Thanks, > -John > > On Mon, Dec 5,

Re: same keys appearing in state stores on different pods when using branches in kafka streams

. John, can you provide more details on the groupBy option? On Tue, Nov 29, 2022 at 12:24 PM Pushkar Deole wrote: > Hi John, > > I am not sure I understood it correctly, even with branching that uses a > different state store, the key of incoming event is still the same, so

Re: same keys appearing in state stores on different pods when using branches in kafka streams

to make the > groupBy behave the way you expected. If that’s interesting to you, let us > know and we can give you some pointers! > > I hope this helps, > John > > On Wed, Nov 23, 2022, at 05:36, Pushkar Deole wrote: > > Hi All, > > > > I have a stream ap

same keys appearing in state stores on different pods when using branches in kafka streams

Hi All, I have a stream application that creates 2 branches. Each branch includes a state store where the status field of the kafka message determines the branch, and therefore the state store used: Status OPEN = State store name totals Status CLOSED = State store name records I’m seeing

Re: Kafka consumer ensure offsets committed before partition rebalance to avoid duplicates

no revoke) -> > rebalancing -> received new assignment -> revoke partitions not owned > anymore > > So you can see, in cooperative protocol, since it didn't revoke any > partition before rebalancing, it might fetch more data after offset > commits. > > Hope th

Re: Kafka consumer ensure offsets committed before partition rebalance to avoid duplicates

ug that we're still working on: > KAFKA-14224 <https://issues.apache.org/jira/browse/KAFKA-14224> > > Thank you. > Luke > > On Wed, Sep 14, 2022 at 3:09 PM Pushkar Deole > wrote: > > > Hi All, > > > > I am hosting kafka consumers inside microservic

Kafka consumer ensure offsets committed before partition rebalance to avoid duplicates

Hi All, I am hosting kafka consumers inside microservice hosted as kubernetes pods, 3 consumers in a consumer group. There is a requirement to add auto-scaling where there will be a single pod which will be auto-scaled out or scaled-in based on the load on microservice. So, 1 pod can be scaled

Re: source topic partitions not assigned evenly to kafka stream threads

@Matthias Can you help with this since I remember having conversation with you in the past on this topic wherein it was mentioned that partition assignment to stream task logic might change in future releases On Mon, Jun 13, 2022, 11:05 Pushkar Deole wrote: > Hi, > > I have a mic

source topic partitions not assigned evenly to kafka stream threads

Hi, I have a microservice hosting kafka streams application. I have 3 instances of microservice hosted in 3 pods, each is having 2 kafka stream threads, thus total 6 stream threads as part of consumer group. There are 3 source topics: A, B, C each having 12, 6, 6 partitions respectively i.e.

Re: kafka producer exception due to TimeoutException: Expiring records for topic 120000ms has passed since batch creation

onfigured for? > > Cheers, > > Liam > > On Tue, 5 Apr 2022 at 05:39, Pushkar Deole wrote: > > > Hi All, > > > > We are intermittently seeing KafkaProducerException. The nested exception > > is as below: > > > > org.springframework.kafka

kafka producer exception due to TimeoutException: Expiring records for topic 120000ms has passed since batch creation

Hi All, We are intermittently seeing KafkaProducerException. The nested exception is as below: org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: Expiring 10 record(s) for analytics.mpe.passthrough-0:12

if kafka producer client app crashes, does kafka server cleanup server side resources

Hi All, I am wondering what would happen if the producer app crashes without calling producer.close, in this case would kafka server take care of cleaning up resources allocated on kafka server/broker for those producers? Or whether those resources be leaked on server side and How does kafka

Re: uneven distribution of events across kafka topic partitions for small number of unique keys

t; Regards, > Dave > > > On Nov 21, 2021, at 7:38 AM, Pushkar Deole wrote: > > > > Thanks Luke, I am sure this problem would have been faced by many others > > before so would like to know if there are any existing custom algorithms > > that can be reused, > >

Re: uneven distribution of events across kafka topic partitions for small number of unique keys

gt; partition-2... etc. > > Thank you. > Luke > > > On Sat, Nov 20, 2021 at 2:55 PM Pushkar Deole > wrote: > > > Hi All, > > > > We are experiencing some uneven distribution of events across topic > > partitions for a small set of unique keys: followi

uneven distribution of events across kafka topic partitions for small number of unique keys

Hi All, We are experiencing some uneven distribution of events across topic partitions for a small set of unique keys: following are the details: 1. topic with 6 partitions 2. 8 unique keys used to produce events onto the topic Used 'key' based partitioning while producing events onto the above

Producer Timeout issue in kafka streams task

Hi All, I am getting below issue in streams application. Kafka cluster is a 3 broker cluster (v 2.5.1) and I could see that 2 of the 3 brokers restarted at the same time when below exception occurred in streams application so I can relate below exception to those brokers restarts. However, what

does kafka provide REST API for producing events to topic?

Hi All, I am trying to load test our application that is a consumer of a kafka topic. I want to load test the application wherein my test tool (jmeter script) is making a connection to kafka broker and then producing events onto topic through producer API. This requires that the test client has

Re: kafka streams commit.interval.ms for at-least-once too high

essing and offset commits of events that are already consumed and > being > > processed for streams instance? > > If a rebalance/shutdown is triggered, Kafka Streams will stop processing > new records and just finish processing all in-flight records. > Afterwards, a commit happens right

Re: kafka streams commit.interval.ms for at-least-once too high

mitted each 30 > seconds by default. This continuous output is also the reason why there > is not latency impact for at-least-once using a longer commit interval. > > Beside an impact on latency, there is also a throughput impact. Using a > longer commit interval provides higher throughput.

kafka streams commit.interval.ms for at-least-once too high

Hi All, I am looking into the commit.interval.ms in kafka streams which says that it is the time interval at which streams would commit offsets to source topics. However for exactly-once guarantee, default of this time is 100ms whereas for at-least-once it is 3ms (i.e. 30sec) Why is there

Re: Redis as state store

ple -- why > not just have > one application write the results to an output topic, and then let the > downstream application > read in that topic as a table? You should be able to have full control over > things that way > > On Sun, Mar 21, 2021 at 5:58 AM Pushkar Deole >

Re: Is exactly-once possible with kafka consumer-producer ?

> Kafka, that said. Wherever Kafka terminates you'd have to code it yourself. > > > > On Sat, Jul 17, 2021 at 2:01 AM Pushkar Deole > wrote: > > > Chris, > > > > I am not sure how this solves the problem scenario that we are > experiencing > > in cus

Re: Is exactly-once possible with kafka consumer-producer ?

legit to > me ;) > > Chris > > > > On Fri, Jul 16, 2021 at 10:48 Pushkar Deole wrote: > > > Thanks Chris for the response! > > > > The current application is quite evolved and currently using > > > > consumer-producer model described above an

Re: Is exactly-once possible with kafka consumer-producer ?

. But unless you are using Streams, you > chain those together yoursef. Would kafka streams work for the operation > you’re looking to do? > > Best, > Chris > > On Fri, Jul 16, 2021 at 08:30 Pushkar Deole wrote: > > > Hi All, > > > > > > > &

Is exactly-once possible with kafka consumer-producer ?

Hi All, I am using a normal kafka consumer-producer in my microservice, with a simple model of consume from source topic -> process the record -> produce on destination topic. I am mainly looking for exactly-once guarantee wherein the offset commit to consumed topic and produce on destination

Re: Confluent's parallel consumer

events from the > broker? > > Please could you elaborate/clarify? > > On Thu, Jul 15, 2021 at 10:00 AM Pushkar Deole > wrote: > > > Well... with key-level ordering, i am mainly concerned about event loss, > if > > any, in below mentioned scenario: > > > >

Re: Confluent's parallel consumer

based ordering with > synchronous commits to start with. > > > > On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole > wrote: > > > Hi All, and Antony (author of below article) > > > > i came across this article which seemed interesting: Introducing > > Confluent’

Confluent's parallel consumer

Hi All, and Antony (author of below article) i came across this article which seemed interesting: Introducing Confluent’s Parallel Consumer Message Processing Client I would like to use the key-level

Re: does kafka streams guarantee EOS with external producer used in application?

> `context.commit()` only schedules an earlier commit, but after the call > returned, no commit happened yet (just a request to commit asap was > registered). > > > -Matthias > > On 7/14/21 12:00 AM, Pushkar Deole wrote: > > If async-writes are used either with man

Re: does kafka streams guarantee EOS with external producer used in application?

use EOS, you cannot use your own producer, but you need > to use a "sink node" (via `addSink()` or `to()`). > > For at-least-once, if you use sync-writes, you should still get > at-least-once guarantees. > > -Matthias > > On 7/9/21 4:12 AM, Pushkar Deole wrote: > &g

Re: does kafka streams guarantee EOS with external producer used in application?

Matthias, Do you have any inputs on above queries? On Wed, Jun 30, 2021 at 7:15 PM Pushkar Deole wrote: > Hi, > > Our application uses kafka streams that reads from a source topic, does > processing on records and produces processed record on destination topic > through the

does kafka streams guarantee EOS with external producer used in application?

Hi, Our application uses kafka streams that reads from a source topic, does processing on records and produces processed record on destination topic through the use of external producer i.e. the producer created via kafka producer API. Does this model still guarantee exactly once semantic or it

does consumer thread wait for producer to return (synchronous) in normal consume-process-produce topology? And how it is handled in streams?

Hi, I am trying to understand few things: in a normal consumer-process-produce topology, consumer is polling records, then process each and then gives to producer to produce on destination topic. In this case, is the 'produce' a synchronous call i.e does it happen in the same consumer thread or

Re: does kafka support reducing topic partitions on the fly?

tation/, "Kafka does > not currently support reducing the number of partitions for a topic.". > Andrew > > On Wed, May 19, 2021 at 10:17 AM Pushkar Deole > wrote: > > > Hi All, > > > > I have a question around reducing topic partitions on kafka. Currently,

does kafka support reducing topic partitions on the fly?

Hi All, I have a question around reducing topic partitions on kafka. Currently, all topics are 6 partitions. I want to reduce it to 3 for some of the topics. Is this supported by kafka without deleting the topic?

Re: kafka metric to monitor for consumer FETCH using disk caching and not going to disk

fka), you can determine if you are > effectively hitting the disk or not. > > [1] > https://cwiki.apache.org/confluence/display/KAFKA/KIP-551%3A+Expose+disk+read+and+write+metrics > > Thanks, > Alexandre > > Le sam. 15 mai 2021 à 05:49, Pushkar Deole a écrit > : > > >

kafka metric to monitor for consumer FETCH using disk caching and not going to disk

Hi All, is there any metric that I can use to check whether the memory allocated for kafka is sufficient for the given load on the brokers and whether kafka is optimally making use of page cache for consumer fetch reads which are not going to disk for each read slowing down the overall consumer

Re: Redis as state store

er to > take an occasional > performance hit to keep their app lightweight. > > Anyways, these are just some thoughts on how to improve the current > situation. Maybe there are > even more options to address this problem which haven't been considered > yet. Let us know if

Re: Redis as state store

//issues.apache.org/jira/browse/KAFKA-12475?focusedCommentId=17305191=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17305191 > > > > Does that address your question? > > On Fri, Mar 19, 2021 at 10:14 AM Pushkar Deole > wrote: > > > Hello Sophie, > >

Re: Redis as state store

t; > was > > > testing it with EOS enabled. (forcing crashes to occur and checking > that > > > the result of my aggregation was still accurate) I was unable to cause > > > inconsistent data in the mongo store (which is good!), though of > course I > >

Re: does Kafka exactly-once guarantee also apply to kafka state stores?

lementation". > > For this case, you cannot get exactly-once. > > There are actually ideas to improve the implementation to support the > case you describe, but there is no timeline for this change yet. Not > even sure if there is already a Jira ticket... > > > -Matthias

Re: Redis as state store

Bruno, i tried to explain this in 'kafka user's language through above mentioned scenario, hope i put it properly -:) and correct me if i am wrong On Mon, Mar 15, 2021 at 7:23 PM Pushkar Deole wrote: > This is what I understand could be the issue with external state store: > > kaf

Re: Redis as state store

C > > On Sun, Mar 14, 2021 at 4:58 PM Parthasarathy, Mohan > wrote: > > > Thanks for the responses. In the worst case, I might have to keep both > > rocksdb for local store and keep an external store like Redis. > > > > -mohan > > > > > > On 3

Re: Redis as state store

Another issue with 3rd party state stores could be violation of exactly-once guarantee provided by kafka streams in the event of a failure of streams application instance. Kafka provides exactly once guarantee for consumer -> process -> produce through transactions and with the use of state store

Re: options for kafka cluster backup?

nect s3 sink. You'd have to write > > some disaster-recovery code to restore lost data from s3 into Kafka. > > > > https://www.confluent.io/blog/apache-kafka-to-amazon-s3-exactly-once/ > > > > There are many other sinks available, but s3 might be a reasonable choice > > for backup

Re: options for kafka cluster backup?

ferent rack/sites awareness to avoid data lose, Confluent's > Replicator or MirrorMaker are for copying data from one cluster to another > usually in different dc / regions, If I am not mistaken > > בתאריך יום ו׳, 5 במרץ 2021, 11:21, מאת Pushkar Deole ‏< > pdeole2...@gmail.co

Re: options for kafka cluster backup?

> ref: https://kafka.apache.org/documentation/#georeplication-mirrormaker > > Thanks. > Luke > > On Fri, Mar 5, 2021 at 1:50 PM Pushkar Deole wrote: > > > Hi All, > > > > I was looking for some options to backup a running kafka cluster, for > > disas

options for kafka cluster backup?

Hi All, I was looking for some options to backup a running kafka cluster, for disaster recovery requirements. Can someone provide what are the available options to backup and restore a running cluster in case the entire cluster goes down? Thanks..

Re: does Kafka exactly-once guarantee also apply to kafka state stores?

lementation". > > For this case, you cannot get exactly-once. > > There are actually ideas to improve the implementation to support the > case you describe, but there is no timeline for this change yet. Not > even sure if there is already a Jira ticket... > > > -Matthias

Re: does Kafka exactly-once guarantee also apply to kafka state stores?

topic to back up the state store, then it may > >> not lose data. > >> > >> Also, if the third party store is not "kafka community certified" (or > not > >> well-maintained), it may have chances to lose data (in different ways). > >> > &

Re: does Kafka exactly-once guarantee also apply to kafka state stores?

: > The physical store behind "state store" is change-log kafka topic. In > Kafka stream, if something fails in the middle, the "state store" is > restored back to the state before the event happens at the first step / > beginning of the stream. > > > > On 2020/

does Kafka exactly-once guarantee also apply to kafka state stores?

Hi All, We use Kafka streams and may need to use exactly-once configuration for some of the use cases. Currently, the application uses either local or global state store to store state. So, the application will consume events from source kafka topic, process the events, for state stores it will

Re: KIP to Gracefully handle timeout exception on kafka streams

eption you hit, it's `max.block.ms` -- you might also look > into `default.api.timeout.ms`. > > In general, the relevant configs are documented in the JavaDocs of the > corresponding client method. > > > -Matthias > > On 11/20/20 9:11 PM, Pushkar Deole wrote: > > Th

Re: multi-threaded consumer configuration like stream threads?

he other hand > injecting backoff-delay would require consideration about how to not block > subsequent messages. > > (FYI, Decaton provides retry-queueing with backoff out-of-the box. :) > https://github.com/line/decaton/blob/master/docs/retry-queueing.adoc) > > 2020年11月24日(火)

Re: multi-threaded consumer configuration like stream threads?

gt; > > 2020年11月24日(火) 1:16 Pushkar Deole : > > > Thanks Liam! > > We don't have a requirement to maintain order of processing for events > even > > within a partition. Essentially, these are events for various accounts > > (customers) that we want to suppor

Re: multi-threaded consumer configuration like stream threads?

> >- *CON*: Manually committing the position becomes harder as it > >requires that all threads co-ordinate to ensure that processing is > complete > >for that partition. > > > > There are many possible variations on this approach. For example each > > proc

Re: KIP to Gracefully handle timeout exception on kafka streams

throws a timeout exception. > > There is however https://issues.apache.org/jira/browse/KAFKA-8803 that > you could have hit, too. > > > -Matthias > > On 11/18/20 7:05 AM, Pushkar Deole wrote: > > Matthias, > > > > We recently ran into an issue where kafka brokers upg

KIP to Gracefully handle timeout exception on kafka streams

Matthias, We recently ran into an issue where kafka brokers upgraded (i guess it was rolling update) for Aiven business plan 4 to plan 8. This involves change to cpu, memory and storage for each broker. Since this should be rolling upgrade, we expected services to survive, however in one service

multi-threaded consumer configuration like stream threads?

Hi, Is there any configuration in kafka consumer to specify multiple threads the way it is there in kafka streams? Essentially, can we have a consumer with multiple threads where the threads would divide partitions of topic among them?

Stream stopped running due to TimeoutException: Timeout expired after 60000milliseconds while awaiting InitProducerId

Hi All, Recently we moved kafka to azure lab and we are seeing these exceptions quite often. Strangely only some of the stream apps get this error and stop working, while other stream app run fine. Came across this issue on kafka issues list https://issues.apache.org/jira/browse/KAFKA-8803 and

Re: Kafka streams parallelism - why not separate stream task per partition per input topic

analytics-event-filter-StreamThread-12-consumer On Fri, Oct 9, 2020 at 8:20 AM Pushkar Deole wrote: > I looked at the task assignment and it looked random for some threads: > e.g. i have 3 topics 24 partitions each and have 3 instances of > application. So, each instance assigned 8 partitions

Re: Kafka streams parallelism - why not separate stream task per partition per input topic

? > > -Matthias > > On 10/6/20 10:13 AM, Pushkar Deole wrote: > > So, what do you suggest to address the topic C with lesser traffic? > Should > > we create a separate StreamBuilder and build a separate topology for > topic > > C so we can configure number o

Re: Kafka streams parallelism - why not separate stream task per partition per input topic

not able to know if one topic has less traffic than others > and thus must blindly assume (what is of course a simplification) that > all topics have the same traffic. We only consider the difference > between stateless and stateful tasks atm. > > -Matthias > > On 10/6/20 3:57 A

Re: Kafka streams parallelism - why not separate stream task per partition per input topic

cuting one task. > > > -Matthias > > On 9/4/20 1:30 AM, Pushkar Deole wrote: > > Matthias, > > > > Let's say we have independent sub topologies like: in this case, will the > > streams create tasks equal to the total number of partitions from topicA > > a

started getting TopologyException: Invalid topology after moving to streams-2.5.1

Hi All, After moving to kafka-streams-2.5.1 version, one of our services started failing with below exception. Any idea what this is about and why it was passing with 2.5.0? Any changes made in 2.5.1 that is breaking this? Exception in thread "main"

Re: Kafka streams - how to handle application level exception in event processing

links Gilles posted. > > Best, > Bruno > > On 22.09.20 10:51, Pushkar Deole wrote: > > Thank you Gilles..will take a look.. > > > > Bruno, thanks for your elaborate explanation as well... however it > > basically exposes my application to certain issues..

Re: Kafka streams - how to handle application level exception in event processing

issing in Kafka Streams, you are very > > welcome to search through the tickets in > > https://issues.apache.org/jira/projects/KAFKA/issues and comment on > > tickets that would solve your issue or create a new one if you cannot > > find any. > > > > Best,

Re: Kafka streams - how to handle application level exception in event processing

ing is emitted. > > Best, > Bruno > > On 21.09.20 14:21, Pushkar Deole wrote: > > Bruno, > > > > 1. the loading of topic mapped to GlobalKTable is done by some other > > service/application so when my application starts up, it will just sync a > > GlobalKTable aga

kafka schema registry - some queries and questions

Hi All, Wanted to understand a bit more on the schema registry provided by confluent. Following are the queries: 1. Is the schema registry provided by confluent over the top of Apache Kafka? 2. If a managed kafka service is used in cloud e.g. say Aiven Kafka, then does the schema registry

Re: Kafka streams - how to handle application level exception in event processing

this caveats do not apply > to your use case. > > Best, > Bruno > > > On 21.09.20 13:45, Pushkar Deole wrote: > > Say the application level exception is named as : > > MeasureDefinitionNotAvaialbleException > > > > What I am trying to achieve is: in ab

Re: Kafka streams - how to handle application level exception in event processing

of current event to complete. If I just catch the MeasureDefinitionNotAvaialbleException in processor and log it then the stream will proceed onto next event considering the current event processing got successful right? On Mon, Sep 21, 2020 at 5:11 PM Pushkar Deole wrote: > It is not a ka

Re: Kafka streams - how to handle application level exception in event processing

no > > > [1] https://kafka.apache.org/documentation/#max.task.idle.ms > [2] https://kafka.apache.org/documentation/#commit.interval.ms > > On 21.09.20 12:38, Pushkar Deole wrote: > > Hi, > > > > I would like to know how to handle following scenarios while processing > &

Kafka streams - how to handle application level exception in event processing

Hi, I would like to know how to handle following scenarios while processing events in a kafka streams application: 1. the streams application needs data from a globalKtable which loads it from a topic that is populated by some other service/application. So, if the streams application starts

Re: Kafka stream error - Consumer is not subscribed to any topics or assigned any partitions

Sophie, one more question: will just upgrading kafka-streams jar to 2.5.1 will work or we need to other jars also to be upgraded to 2.5.1 e.g. kafka-clients etc. ? On Mon, Sep 14, 2020 at 7:16 PM Pushkar Deole wrote: > Thanks Sophie... if we are just creating a global state st

Re: Kafka stream error - Consumer is not subscribed to any topics or assigned any partitions

stener, you'll want to > use 2.5.1 > > It's always a good idea to upgrade when a new bugfix version is released > anyway > > On Fri, Sep 11, 2020 at 5:15 AM Pushkar Deole > wrote: > > > Hi All, > > > > I upgraded from Kafka streams 2.4 to 2.5.0 and

Kafka stream error - Consumer is not subscribed to any topics or assigned any partitions

Hi All, I upgraded from Kafka streams 2.4 to 2.5.0 and one of the applications suddenly stopped working with the error message: Exception in thread "DsiApplication-0fcde033-dab2-431c-9d82-76e85fcb4c91-StreamThread-1" java.lang.IllegalStateException: Consumer is not subscribed to any topics or

Re: Error in kafka streams: The producer attempted to use a producer id which is not currently assigned to its transactional id

goes into production. On Thu, Sep 10, 2020 at 2:20 AM Matthias J. Sax wrote: > Well, it's for sure EOS related, but it seems to be a different root cause. > > I am not aware of any related bug. > > -Matthias > > > On 9/9/20 4:29 AM, Pushkar Deole wrote: > > Hi

Re: Error in kafka streams: The producer attempted to use a producer id which is not currently assigned to its transactional id

:4,"stack_trace":"org.apache.kafka.common.errors.InvalidPidMappingException: The producer attempted to use a producer id which is not currently assigned to its transactiona l id.\n"} On Sat, May 9, 2020 at 12:49 AM Matthias J. Sax wrote: > >> So does this issue

Re: Kafka streams parallelism - why not separate stream task per partition per input topic

e is actually a related > ticket: https://issues.apache.org/jira/browse/KAFKA-9282 > > > Hope this helps. > -Matthias > > On 9/2/20 11:09 PM, Pushkar Deole wrote: > > Hi, > > > > I came across articles where it is explained how parallelism is hand

Kafka streams parallelism - why not separate stream task per partition per input topic

Hi, I came across articles where it is explained how parallelism is handled in kafka streams. This is what I collected: When the streams application is reading from multiple topics, the topic with maximum number of partitions is considered for instantiating stream tasks so 1 task is instantiated

Re: kafka per topic metrics

> Also recommend chapter 10 of Kafka: The Definitive Guide, it covers the > metrics really well. > > Cheers, > > Liam Clarke-Hutchinson > > On Thu, 16 Jul. 2020, 5:05 pm Pushkar Deole, wrote: > > > Thanks Liam... > > Few questions: in your pattern

Re: kafka per topic metrics

bean inside the JVM > and > > then exports it via any pre-existing Prometheus registry. You might want > to > > then use a Prometheus aggregating rule to collate all the per-consumer > apps > > into a per-consumer-group metric, unless the per consumer granularity is > of

Re: kafka per topic metrics

grating them into prometheus. > > Hope this helps. > Greetings, > Claudia > > -Ursprüngliche Nachricht- > Von: Pushkar Deole > Gesendet: Mittwoch, 15. Juli 2020 09:07 > An: users@kafka.apache.org > Betreff: Re: kafka per topic metrics > > We are using pro

Re: kafka per topic metrics

yours. > > Thanks and Regards, > Rohit > > On Wed, Jul 15, 2020, 10:33 Pushkar Deole wrote: > > > Hi All, > > > > Any inputs as to how the kafka consumer and producer metrics can be > hooked > > up to a monitoring system such as prometheus ? > > >

Re: kafka per topic metrics

Hi All, Any inputs as to how the kafka consumer and producer metrics can be hooked up to a monitoring system such as prometheus ? On Tue, Jul 14, 2020 at 4:22 PM Pushkar Deole wrote: > i did find these metrics from confluent docs: however how can i get this > metric ? is it available at

Re: kafka per topic metrics

. On Tue, Jul 14, 2020 at 3:27 PM Pushkar Deole wrote: > Hi All, > > Need some help on kafka metrics, i am interested in certain metrics e.g. i > need to know the number of records published on a particular topic and > number of records consumed from that topic by a specific consumer g

  1   2   >