Boyang Chen created KAFKA-9645:
----------------------------------
Summary: Records could not find corresponding partition/task
Key: KAFKA-9645
URL: https://issues.apache.org/jira/browse/KAFKA-9645
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.6.0
Reporter: Boyang Chen
Assignee: Boyang Chen
We could be hitting the illegal state when stream kicks off a rebalance with
all tasks closed:
```
[2020-03-03T18:36:09-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04
02:36:09,105] WARN
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1]
stream-thread
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] Detected
that the thread is being fenced. This implies that this thread missed a
rebalance and dropped out of the consumer group. Will close out all assigned
tasks and rejoin the consumer group.
(org.apache.kafka.streams.processor.internals.StreamThread)
[2020-03-03T18:36:09-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04
02:36:09,105] INFO
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1]
[Consumer
clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-restore-consumer,
groupId=null] Subscribed to partition(s):
stream-soak-test-KSTREAM-AGGREGATE-STATE-STORE-0000000049-changelog-2
(org.apache.kafka.clients.consumer.KafkaConsumer)
[2020-03-03T18:36:10-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04
02:36:09,286] INFO
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1]
[Producer
clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-1_1-producer,
transactionalId=stream-soak-test-1_1] Closing the Kafka producer with
timeoutMillis = 9223372036854775807 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2020-03-03T18:36:10-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04
02:36:09,287] INFO
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1]
stream-thread
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] task
[1_1] Closed dirty (org.apache.kafka.streams.processor.internals.StreamTask)
[2020-03-03T18:36:10-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04
02:36:09,287] INFO
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1]
[Consumer
clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-restore-consumer,
groupId=null] Unsubscribed all topics or patterns and assigned partitions
(org.apache.kafka.clients.consumer.KafkaConsumer)
[2020-03-03T18:36:10-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04
02:36:09,290] INFO
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1]
[Producer
clientId=stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1-3_2-producer,
transactionalId=stream-soak-test-3_2] Closing the Kafka producer with
timeoutMillis = 9223372036854775807 ms.
(org.apache.kafka.clients.producer.KafkaProducer)
[2020-03-03T18:36:10-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04
02:36:09,292] INFO
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1]
stream-thread
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] task
[3_2] Closed dirty (org.apache.kafka.streams.processor.internals.StreamTask)
[2020-03-03T18:36:10-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04
02:36:09,293] ERROR
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1]
stream-thread
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] Unable
to locate active task for received-record partition node-name-repartition-1.
Current tasks: TaskManager
[2020-03-03T18:36:10-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) >
MetadataState:
[2020-03-03T18:36:10-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) > Tasks:
(org.apache.kafka.streams.processor.internals.StreamThread)
[2020-03-03T18:36:10-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04
02:36:09,293] ERROR
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1]
stream-thread
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1]
Encountered the following exception during processing and the thread is going
to shut down: (org.apache.kafka.streams.processor.internals.StreamThread)
[2020-03-03T18:36:10-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog)
java.lang.NullPointerException: Task was unexpectedly missing for partition
node-name-repartition-1
at
org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:984)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:820)
```
We should be more careful in this case by avoiding processing data when no
corresponding task is found.
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:751)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:725)
[2020-03-03T18:36:10-08:00]
(streams-soak-trunk-eos_soak_i-050294ea2392cf355_streamslog) [2020-03-04
02:36:09,294] INFO
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1]
stream-thread
[stream-soak-test-e639a3a1-bd9d-49e0-896e-df9fe0cd63db-StreamThread-1] State
transition from RUNNING to PENDING_SHUTDOWN
(org.apache.kafka.streams.processor.internals.StreamThread)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
