Ivan Atanasov created KAFKA-6212:
------------------------------------
Summary: Kafka Streams - Incorrect partition rebalancing
Key: KAFKA-6212
URL: https://issues.apache.org/jira/browse/KAFKA-6212
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 0.10.0.1
Environment: Linux
Reporter: Ivan Atanasov
Trying to use streaming with version 0.10.0.1 of kafka but it is not working
how I'd expect. I realize that this is a fairly old version now but it is what
we are running and are not in a position to upgrade right now.
The particular problem I am having is when an extra instance of the streaming
app is run using the same application ID. What seems to happen is the newly
introduced instance takes half of the partitions available, which is expected
but the original instance drops all the partitions it was reading from.
therefore from then on, data is only read from half the partitions.
Strangely offsets are still being committed for the other partitions but the
data from them is not consumed as expected.
My topology is very simple for now, all it does is a print of the message. Also
I have tried making both instances use different client IDs and state
directories.
Is this a known bug in 0.10.0.1?
*Logs Below:*
Instance 1:
{quote}[2017-11-15 10:41:41,597] INFO [StreamThread-1] Setting newly assigned
partitions [rawEvents-5, rawEvents-6, rawEvents-3, rawEvents-4, rawEvents-9,
rawEvents-7, rawEvents-8, rawEvents-1, rawEvents-2, rawEvents-0] for group
kafka-stream-test
(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2017-11-15 10:41:41,616] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_0
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,645] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_1
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,645] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_2
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_3
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_4
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,646] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_5
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_6
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_7
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,647] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_8
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:41:41,648] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_9
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:42:08,682] INFO [StreamThread-1] Revoking previously assigned
partitions [rawEvents-5, rawEvents-6, rawEvents-3, rawEvents-4, rawEvents-9,
rawEvents-7, rawEvents-8, rawEvents-1, rawEvents-2, rawEvents-0] for group
kafka-stream-test
(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2017-11-15 10:42:08,682] INFO [StreamThread-1] Removing a task 0_0
(org.apache.kafka.streams.processor.internals.StreamThread){quote}
Instance 2:
{quote}[2017-11-15 10:42:08,827] INFO [StreamThread-1] Successfully joined
group kafka-stream-test with generation 2
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
[2017-11-15 10:42:08,829] INFO [StreamThread-1] Setting newly assigned
partitions [rawEvents-5, rawEvents-3, rawEvents-1, rawEvents-2, rawEvents-0]
for group kafka-stream-test
(org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2017-11-15 10:42:08,840] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_0
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:42:08,869] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_1
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:42:08,870] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_2
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:42:08,871] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_3
(org.apache.kafka.streams.processor.internals.StreamTask)
[2017-11-15 10:42:08,872] INFO [StreamThread-1] Creating restoration consumer
client for stream task #0_5
(org.apache.kafka.streams.processor.internals.StreamTask){quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
