Re: Kafka Streams reaching ERROR state during rolling upgrade / restart of brokers
Matthias It happened again yesterday during another rolling update. The first error log I can see on the client side is below. It was there in PENDING_ERROR state for sometime and then went into ERROR state. Caused by: java.lang.IllegalStateException: KafkaStreams is not running. State is PENDING_ERROR. at org.apache.kafka.streams.KafkaStreams.validateIsRunningOrRebalancing(KafkaStreams.java:381) at org.apache.kafka.streams.KafkaStreams.queryMetadataForKey(KafkaStreams.java:1663) at org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$2(InteractiveQueryService.java:227) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.HashMap$KeySpliterator.tryAdvance(HashMap.java:1728) at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129) at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:647) at org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$3(InteractiveQueryService.ja On Tue, Oct 3, 2023 at 8:50 AM Matthias J. Sax wrote: > I did mean client side... If KS goes into ERROR state, it should log > the reason. > > If the logs are indeed empty, try to register an > uncaught-exception-handler via > > KafkaStreamssetUncaughtExceptionHandler(...) > > > -Matthias > > On 10/2/23 12:11 PM, Debraj Manna wrote: > > Are you suggesting to check the Kafka broker logs? I do not see any other > > errors logs on the client / application side. > > > > On Fri, 29 Sep, 2023, 22:01 Matthias J. Sax, wrote: > > > >> In general, Kafka Streams should keep running. > >> > >> Can you inspect the logs to figure out why it's going into ERROR state > >> to begin with? Maybe you need to increase/change some timeouts/retries > >> configs. > >> > >> The stack trace you shared, is a symptom, but not the root cause. > >> > >> -Matthias > >> > >> On 9/21/23 12:56 AM, Debraj Manna wrote: > >>> I am using Kafka broker 2.8.1 (from AWS MSK) with Kafka clients and > Kafka > >>> stream 3.5.1. > >>> > >>> I am observing that whenever some rolling upgrade is done on AWS MSK > our > >>> stream application reaches an error state. I get the below exception on > >>> trying to query the state store > >>> > >>> caused by: java.lang.IllegalStateException: KafkaStreams is not > running. > >>> State is ERROR. > >>> at > >>> > >> > org.apache.kafka.streams.KafkaStreams.validateIsRunningOrRebalancing(KafkaStreams.java:381) > >>> at > >>> > >> > org.apache.kafka.streams.KafkaStreams.queryMetadataForKey(KafkaStreams.java:1663) > >>> at > >>> > >> > org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$2(InteractiveQueryService.java:227) > >>> > >>> Can someone let me know what the recommended way we can keep the stream > >>> application running whenever some rolling upgrade/restart of brokers is > >>> done in the background? > >>> > >> > > >
Re: Kafka Streams reaching ERROR state during rolling upgrade / restart of brokers
I did mean client side... If KS goes into ERROR state, it should log the reason. If the logs are indeed empty, try to register an uncaught-exception-handler via KafkaStreamssetUncaughtExceptionHandler(...) -Matthias On 10/2/23 12:11 PM, Debraj Manna wrote: Are you suggesting to check the Kafka broker logs? I do not see any other errors logs on the client / application side. On Fri, 29 Sep, 2023, 22:01 Matthias J. Sax, wrote: In general, Kafka Streams should keep running. Can you inspect the logs to figure out why it's going into ERROR state to begin with? Maybe you need to increase/change some timeouts/retries configs. The stack trace you shared, is a symptom, but not the root cause. -Matthias On 9/21/23 12:56 AM, Debraj Manna wrote: I am using Kafka broker 2.8.1 (from AWS MSK) with Kafka clients and Kafka stream 3.5.1. I am observing that whenever some rolling upgrade is done on AWS MSK our stream application reaches an error state. I get the below exception on trying to query the state store caused by: java.lang.IllegalStateException: KafkaStreams is not running. State is ERROR. at org.apache.kafka.streams.KafkaStreams.validateIsRunningOrRebalancing(KafkaStreams.java:381) at org.apache.kafka.streams.KafkaStreams.queryMetadataForKey(KafkaStreams.java:1663) at org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$2(InteractiveQueryService.java:227) Can someone let me know what the recommended way we can keep the stream application running whenever some rolling upgrade/restart of brokers is done in the background?
Re: Kafka Streams reaching ERROR state during rolling upgrade / restart of brokers
Are you suggesting to check the Kafka broker logs? I do not see any other errors logs on the client / application side. On Fri, 29 Sep, 2023, 22:01 Matthias J. Sax, wrote: > In general, Kafka Streams should keep running. > > Can you inspect the logs to figure out why it's going into ERROR state > to begin with? Maybe you need to increase/change some timeouts/retries > configs. > > The stack trace you shared, is a symptom, but not the root cause. > > -Matthias > > On 9/21/23 12:56 AM, Debraj Manna wrote: > > I am using Kafka broker 2.8.1 (from AWS MSK) with Kafka clients and Kafka > > stream 3.5.1. > > > > I am observing that whenever some rolling upgrade is done on AWS MSK our > > stream application reaches an error state. I get the below exception on > > trying to query the state store > > > > caused by: java.lang.IllegalStateException: KafkaStreams is not running. > > State is ERROR. > > at > > > org.apache.kafka.streams.KafkaStreams.validateIsRunningOrRebalancing(KafkaStreams.java:381) > > at > > > org.apache.kafka.streams.KafkaStreams.queryMetadataForKey(KafkaStreams.java:1663) > > at > > > org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$2(InteractiveQueryService.java:227) > > > > Can someone let me know what the recommended way we can keep the stream > > application running whenever some rolling upgrade/restart of brokers is > > done in the background? > > >
Re: Kafka Streams reaching ERROR state during rolling upgrade / restart of brokers
In general, Kafka Streams should keep running. Can you inspect the logs to figure out why it's going into ERROR state to begin with? Maybe you need to increase/change some timeouts/retries configs. The stack trace you shared, is a symptom, but not the root cause. -Matthias On 9/21/23 12:56 AM, Debraj Manna wrote: I am using Kafka broker 2.8.1 (from AWS MSK) with Kafka clients and Kafka stream 3.5.1. I am observing that whenever some rolling upgrade is done on AWS MSK our stream application reaches an error state. I get the below exception on trying to query the state store caused by: java.lang.IllegalStateException: KafkaStreams is not running. State is ERROR. at org.apache.kafka.streams.KafkaStreams.validateIsRunningOrRebalancing(KafkaStreams.java:381) at org.apache.kafka.streams.KafkaStreams.queryMetadataForKey(KafkaStreams.java:1663) at org.springframework.cloud.stream.binder.kafka.streams.InteractiveQueryService.lambda$getHostInfo$2(InteractiveQueryService.java:227) Can someone let me know what the recommended way we can keep the stream application running whenever some rolling upgrade/restart of brokers is done in the background?
