[jira] [Commented] (KAFKA-12818) Memory leakage when kafka connect 2.7 uses directory config provider
[
https://issues.apache.org/jira/browse/KAFKA-12818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348595#comment-17348595
]
Viktor Utkin commented on KAFKA-12818:
--
Hi [~tombentley], what memory usage diagnostics data do you need?
By the way we did tests 2 days under the load and I showed you the result. It's
not a 5 min tests.
> Memory leakage when kafka connect 2.7 uses directory config provider
>
>
> Key: KAFKA-12818
> URL: https://issues.apache.org/jira/browse/KAFKA-12818
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.7.0
> Environment: Azure AKS / Kubernetes v1.20
>Reporter: Viktor Utkin
>Priority: Critical
> Attachments: Screenshot 2021-05-20 at 14.53.05.png
>
>
> Hi, we noticed a Memory leakage problem when kafka connect 2.7 uses directory
> config provider. We've got an OOM in kubernetes environment. K8s kills
> container when limit reached. At same time we've not get any OOM in Java.
> Heap dump did't show us anything interesting.
> JVM config:
> {code:java}
> -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/
> -XX:+UseContainerSupport
> -XX:+OptimizeStringConcat
> -XX:MaxRAMPercentage=75.0
> -XX:InitialRAMPercentage=50.0
> -XX:MaxMetaspaceSize=256M
> -XX:MaxDirectMemorySize=256M
> -XX:+UseStringDeduplication
> -XX:+AlwaysActAsServerClassMachine{code}
>
> Kafka Connect config:
> {code:java}
> "config.providers": "directory"
> "config.providers.directory.class":
> "org.apache.kafka.common.config.provider.DirectoryConfigProvider"{code}
>
> Kubernetes pod resources limits:
> {code:java}
> resources:
> requests:
> cpu: 1500m
> memory: 2Gi
> limits:
> cpu: 3000m
> memory: 3Gi
> {code}
>
> doker image used: confluentinc/cp-kafka-connect:6.1.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-12818) Memory leakage when kafka connect 2.7 uses directory config provider
[
https://issues.apache.org/jira/browse/KAFKA-12818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348590#comment-17348590
]
Tom Bentley commented on KAFKA-12818:
-
Since your limit > request you're going to be at risk of the OOMKiller anyway,
whether or not there's a memory leak. You were exceeding your request even with
the FileConfigProvider.
I find it hard to believe the DirectoryConfigProvider itself is responsible for
somehow using _that much_ (~200m) memory above what we see in the first curve.
It's normally only used once when the config file is parsed and the values
substituted. It doesn't watch the files (even if they do change). So I think we
need better memory usage diagnostics here than just a graph of RSS.
> Memory leakage when kafka connect 2.7 uses directory config provider
>
>
> Key: KAFKA-12818
> URL: https://issues.apache.org/jira/browse/KAFKA-12818
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.7.0
> Environment: Azure AKS / Kubernetes v1.20
>Reporter: Viktor Utkin
>Priority: Critical
> Attachments: Screenshot 2021-05-20 at 14.53.05.png
>
>
> Hi, we noticed a Memory leakage problem when kafka connect 2.7 uses directory
> config provider. We've got an OOM in kubernetes environment. K8s kills
> container when limit reached. At same time we've not get any OOM in Java.
> Heap dump did't show us anything interesting.
> JVM config:
> {code:java}
> -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/
> -XX:+UseContainerSupport
> -XX:+OptimizeStringConcat
> -XX:MaxRAMPercentage=75.0
> -XX:InitialRAMPercentage=50.0
> -XX:MaxMetaspaceSize=256M
> -XX:MaxDirectMemorySize=256M
> -XX:+UseStringDeduplication
> -XX:+AlwaysActAsServerClassMachine{code}
>
> Kafka Connect config:
> {code:java}
> "config.providers": "directory"
> "config.providers.directory.class":
> "org.apache.kafka.common.config.provider.DirectoryConfigProvider"{code}
>
> Kubernetes pod resources limits:
> {code:java}
> resources:
> requests:
> cpu: 1500m
> memory: 2Gi
> limits:
> cpu: 3000m
> memory: 3Gi
> {code}
>
> doker image used: confluentinc/cp-kafka-connect:6.1.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-12818) Memory leakage when kafka connect 2.7 uses directory config provider
[
https://issues.apache.org/jira/browse/KAFKA-12818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348563#comment-17348563
]
Viktor Utkin commented on KAFKA-12818:
--
Hi [~tombentley], I've used the metric container_memory_usage_bytes for this
graph, anyway I didn't understand why file config provider works well instead
of directory.
> Memory leakage when kafka connect 2.7 uses directory config provider
>
>
> Key: KAFKA-12818
> URL: https://issues.apache.org/jira/browse/KAFKA-12818
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.7.0
> Environment: Azure AKS / Kubernetes v1.20
>Reporter: Viktor Utkin
>Priority: Critical
> Attachments: Screenshot 2021-05-20 at 14.53.05.png
>
>
> Hi, we noticed a Memory leakage problem when kafka connect 2.7 uses directory
> config provider. We've got an OOM in kubernetes environment. K8s kills
> container when limit reached. At same time we've not get any OOM in Java.
> Heap dump did't show us anything interesting.
> JVM config:
> {code:java}
> -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/
> -XX:+UseContainerSupport
> -XX:+OptimizeStringConcat
> -XX:MaxRAMPercentage=75.0
> -XX:InitialRAMPercentage=50.0
> -XX:MaxMetaspaceSize=256M
> -XX:MaxDirectMemorySize=256M
> -XX:+UseStringDeduplication
> -XX:+AlwaysActAsServerClassMachine{code}
>
> Kafka Connect config:
> {code:java}
> "config.providers": "directory"
> "config.providers.directory.class":
> "org.apache.kafka.common.config.provider.DirectoryConfigProvider"{code}
>
> Kubernetes pod resources limits:
> {code:java}
> resources:
> requests:
> cpu: 1500m
> memory: 2Gi
> limits:
> cpu: 3000m
> memory: 3Gi
> {code}
>
> doker image used: confluentinc/cp-kafka-connect:6.1.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-12818) Memory leakage when kafka connect 2.7 uses directory config provider
[
https://issues.apache.org/jira/browse/KAFKA-12818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348517#comment-17348517
]
Tom Bentley commented on KAFKA-12818:
-
As you can see from the code, DirectoryConfigProvider doesn't have any state,
so this can't simply be objects being pinned in memory.
How is the memory usage in the graph being calculated (is it process RSS or
some JVM metric)? Assuming the DirConfigProvider was looking at a mounted
directory, was the Secret or ConfigMap for the mount being changed?
> Memory leakage when kafka connect 2.7 uses directory config provider
>
>
> Key: KAFKA-12818
> URL: https://issues.apache.org/jira/browse/KAFKA-12818
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.7.0
> Environment: Azure AKS / Kubernetes v1.20
>Reporter: Viktor Utkin
>Priority: Critical
> Attachments: Screenshot 2021-05-20 at 14.53.05.png
>
>
> Hi, we noticed a Memory leakage problem when kafka connect 2.7 uses directory
> config provider. We've got an OOM in kubernetes environment. K8s kills
> container when limit reached. At same time we've not get any OOM in Java.
> Heap dump did't show us anything interesting.
> JVM config:
> {code:java}
> -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/tmp/
> -XX:+UseContainerSupport
> -XX:+OptimizeStringConcat
> -XX:MaxRAMPercentage=75.0
> -XX:InitialRAMPercentage=50.0
> -XX:MaxMetaspaceSize=256M
> -XX:MaxDirectMemorySize=256M
> -XX:+UseStringDeduplication
> -XX:+AlwaysActAsServerClassMachine{code}
>
> Kafka Connect config:
> {code:java}
> "config.providers": "directory"
> "config.providers.directory.class":
> "org.apache.kafka.common.config.provider.DirectoryConfigProvider"{code}
>
> Kubernetes pod resources limits:
> {code:java}
> resources:
> requests:
> cpu: 1500m
> memory: 2Gi
> limits:
> cpu: 3000m
> memory: 3Gi
> {code}
>
> doker image used: confluentinc/cp-kafka-connect:6.1.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-12818) Memory leakage when kafka connect 2.7 uses directory config provider
[ https://issues.apache.org/jira/browse/KAFKA-12818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17348316#comment-17348316 ] Viktor Utkin commented on KAFKA-12818: -- https://github.com/apache/kafka/pull/9136/files > Memory leakage when kafka connect 2.7 uses directory config provider > > > Key: KAFKA-12818 > URL: https://issues.apache.org/jira/browse/KAFKA-12818 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 2.7.0 > Environment: Azure AKS / Kubernetes v1.20 >Reporter: Viktor Utkin >Priority: Critical > Attachments: Screenshot 2021-05-20 at 14.53.05.png > > > Hi, we noticed a Memory leakage problem when kafka connect 2.7 uses directory > config provider. We've got an OOM in kubernetes environment. At same time > we've not get any OOM in Java. Heap dump did't show us anything interesting. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
