KUMAR created KAFKA-8737:
----------------------------
Summary: TaskMigrated Exception while rebalancing kafka streams
Key: KAFKA-8737
URL: https://issues.apache.org/jira/browse/KAFKA-8737
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 1.0.1, 1.0.0
Environment: 20 partitions
1 topic
8 Streamer service
topic-region-1 9 7841726 8236017
394291
streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/
streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
topic-region-1 15 7421710 7467666 45956
streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/
streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
topic-region-1 19 7737360 8120611
383251
streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer-0276e83d-40b5-4b44-b764-7d29e0dab663/
streams-subscriberstopic-region-1-29d615ed-4243-4b9d-90b7-9c517aa0f2e3-StreamThread-1-consumer
topic-region-1
Reporter: KUMAR
Kafka streams throws following exception while restart of a stream client
service -
o.a.k.s.p.internals.StreamThread.? - stream-thread
[streams-subscriberstopic-region-1-32d968e3-f892-4772-a7a4-6f684d7e43c9-StreamThread-1]
Detected a task that got migrated to another thread. This implies that this
thread missed a rebalance and dropped out of the consumer group. Trying to
rejoin the consumer group now.
org.apache.kafka.streams.errors.TaskMigratedException: Log end offset of
topic-region-1-12 should not change while restoring: old end offset 6286727,
current offset 6380997
Kafka version is 1.0.0 and we have back merged the fix for KIP-6269-
[https://github.com/apache/kafka/pull/4300/files#|https://github.com/apache/kafka/pull/4300/files]
However we observe that there seems to be an issue in rebalance when
"auto.offset.reset" is configured as "latest". Based on log analysis we see
following behavior -
# StreamThread starts a restore consumer
# While Fetching it gets offset out of range
o.a.k.c.consumer.internals.Fetcher.? - [Consumer
clientId=streams-subscriberstopic-region-1-11b2d7fb-11ce-4b0b-a40a-388d3c7b6bc9-StreamThread-1-restore-
consumer, groupId=] Fetch READ_UNCOMMITTED at offset 246431 for partition
topic-region-1-12 returned fetch data (error=OFFSET_OUT_OF_RANGE,
highWaterMark=-1, lastStableOffset = -1, logStartOffset = -1,
abortedTransactions = null, recordsSizeInBytes=0)
# Fetcher tries to reset the offset
# While reset the offset it appears it is changing the offset position and
causing TaskMigrated exception
Above test repeated with "auto.offset.reset" is configured as "earliest" does
not throw any TaskMigrated exception as in earliest case we are not reseting
the restore consumer position.
Please let us know if this is possible and if a fix would be needed for the
offset reset piece when set to latest.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
