Romil Kumar Vasani created KAFKA-7181:
-----------------------------------------
Summary: Kafka Streams State stuck in rebalancing after one of the
StreamThread encounters IllegalStateException
Key: KAFKA-7181
URL: https://issues.apache.org/jira/browse/KAFKA-7181
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 1.1.0
Reporter: Romil Kumar Vasani
One the StreamThread encounters an IllegalStateException and is marked DEAD,
shut down.
The application doesn't spawn a new thread in it's place, the partitions of
that thread are assigned to a different thread and it synchronizes. But the
application is stuck in REBALANCING state, as not all StreamThreads are in
RUNNING state.
Excepted: New thread should come up and after synchronization/rebalancing it
the KafkaStream.State should be RUNNING
Since all the active threads (that are not marked DEAD) are in RUNNING state,
the KafkaStreams.State should be RUNNING
P.S. I am reporting an issue for the first time. If there is more information
needed I can provide.
Below are the logs from the IllegalStateException:
2018-07-18 03:02:27.510 ERROR 1 --- [-StreamThread-2]
o.a.k.s.p.internals.StreamThread : stream-thread
[prd1565.prod.nuke.ops.v1-StreamThread-2] Encountered the following error
during processing:
java.lang.IllegalStateException: No current assignment for partition
consumerGroup-stateStore-changelog-10
at
org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:259)
at
org.apache.kafka.clients.consumer.internals.SubscriptionState.resetFailed(SubscriptionState.java:413)
at
org.apache.kafka.clients.consumer.internals.Fetcher$2.onFailure(Fetcher.java:595)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)
at
org.apache.kafka.clients.consumer.internals.RequestFutureAdapter.onFailure(RequestFutureAdapter.java:30)
at
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:553)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:390)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:293)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1171)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
at
org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1040)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:812)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720)
2018-07-18 03:02:27.511 INFO 1 --- [-StreamThread-2]
o.a.k.s.p.internals.StreamThread : stream-thread
[prd1565.prod.nuke.ops.v1-StreamThread-2] State transition from RUNNING to
PENDING_SHUTDOWN
2018-07-18 03:02:27.511 INFO 1 --- [-StreamThread-2]
o.a.k.s.p.internals.StreamThread : stream-thread
[prd1565.prod.nuke.ops.v1-StreamThread-2] Shutting down
2018-07-18 03:02:27.571 INFO 1 --- [-StreamThread-2]
o.a.k.clients.producer.KafkaProducer : [Producer
clientId=prd1565.prod.nuke.ops.v1-StreamThread-2-producer] Closing the Kafka
producer with timeoutMillis = 9223372036854775807 ms.
2018-07-18 03:02:27.579 INFO 1 --- [-StreamThread-2]
o.a.k.s.p.internals.StreamThread : stream-thread
[prd1565.prod.nuke.ops.v1-StreamThread-2] State transition from
PENDING_SHUTDOWN to DEAD
2018-07-18 03:02:27.579 INFO 1 --- [-StreamThread-2]
o.a.k.s.p.internals.StreamThread : stream-thread [consumerGroup-StreamThread-2]
Shutdown complete
2018-07-18 03:02:27.579 ERROR 1 --- [-StreamThread-2] xxx.xxx.xxx.AppRunner :
Unhandled exception in thread: 43:consumerGroup-StreamThread-2
java.lang.IllegalStateException: No current assignment for partition
consumerGroup-inventoryStore-changelog-10
at
org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:259)
at
org.apache.kafka.clients.consumer.internals.SubscriptionState.resetFailed(SubscriptionState.java:413)
at
org.apache.kafka.clients.consumer.internals.Fetcher$2.onFailure(Fetcher.java:595)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)
at
org.apache.kafka.clients.consumer.internals.RequestFutureAdapter.onFailure(RequestFutureAdapter.java:30)
at
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onFailure(RequestFuture.java:209)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.fireFailure(RequestFuture.java:177)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.raise(RequestFuture.java:147)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:553)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:390)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:293)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
at
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1171)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
at
org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:1040)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:812)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:750)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:720)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
