Di Campo created KAFKA-8165:
-------------------------------
Summary: Streams task causes Out Of Memory after connection and
store restoration
Key: KAFKA-8165
URL: https://issues.apache.org/jira/browse/KAFKA-8165
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.1.0
Environment: Amazon Linux container, on Docker based on
wurstmeister/kafka image.
Reporter: Di Campo
Having a Kafka Streams 2.1 application, when Kafka brokers are stable, the
(largely stateful) application has been consuming ~160 messages per second at a
sustained rate for several hours.
However it started having connection issues to the brokers.
{code:java}
Connection to node 3 (/172.31.36.118:9092) could not be established. Broker may
not be available. (org.apache.kafka.clients.NetworkClient){code}
Also it began showing a lot of these errors:
{code:java}
WARN [Consumer
clientId=stream-processor-81e1ce17-1765-49f8-9b44-117f983a2d19-StreamThread-2-consumer,
groupId=stream-processor] 1 partitions have leader brokers without a matching
listener, including [broker-2-health-check-0]
(org.apache.kafka.clients.NetworkClient){code}
In fact, the _health-check_ topic is in the broker but not consumed by this
topology or used in any way by the Streams application (it is just broker
healthcheck). It does not complain about topics that are actually consumed by
the topology.
Some time after these errors (that appear at a rate of 24 appearances per
second during ~5 minutes), then the following logs appear:
{code:java}
[2019-03-27 15:14:47,709] WARN [Consumer
clientId=stream-processor-81e1ce17-1765-49f8-9b44-117f983a2d19-StreamThread-1-restore-consumer,
groupId=] Connection to node -3 (/ip3:9092) could not be established. Broker
may not be available. (org.apache.kafka.clients.NetworkClient){code}
In between 6 and then 3 lines of "Connection could not be established" error
messages, 3 of these ones slipped in:
[2019-03-27 15:14:47,723] WARN Started Restoration of visitorCustomerStore
partition 15 total records to be restored 17
(com.divvit.dp.streams.applications.monitors.ConsoleGlobalRestoreListener)
... one for each different KV store I have (I still have another KV that does
not appear, and a WindowedStore store that also does not appear).
Then I finally see "Restoration Complete" (using a logging
ConsoleGlobalRestoreListener as in docs) messages for all of my stores. So it
seems it may be fine now to restart the processing.
Three minutes later, some events get processed, and I see an OOM error:
java.lang.OutOfMemoryError: GC overhead limit exceeded
... so given that it usually allows to process during hours under same
circumstances, I'm wondering whether there is some memory leak in the
connection resources or somewhere in the handling of this scenario.
Kafka and KafkaStreams 2.1
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
