[jira] [Commented] (FLINK-5946) Kinesis Producer uses KPL that orphans threads that consume 100% CPU
[ https://issues.apache.org/jira/browse/FLINK-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011874#comment-16011874 ] Tzu-Li (Gordon) Tai commented on FLINK-5946: Great news! Thanks for the info Scott. > Kinesis Producer uses KPL that orphans threads that consume 100% CPU > > > Key: FLINK-5946 > URL: https://issues.apache.org/jira/browse/FLINK-5946 > Project: Flink > Issue Type: Bug > Components: Kinesis Connector >Affects Versions: 1.2.0 >Reporter: Scott Kidder > > It's possible for the Amazon Kinesis Producer Library (KPL) to leave orphaned > threads running after the producer has been instructed to shutdown via the > `destroy()` method. These threads run in a very tight infinite loop that can > push CPU usage to 100%. I've seen this happen on several occasions, though it > does not happen all of the time. Once these threads are orphaned, the only > solution to bring CPU utilization back down is to restart the Flink Task > Manager. > When a KPL producer is instantiated, it creates several threads: one to > execute and monitor the native sender process, and two threads to monitor the > process' stdout and stderr output. It's possible for the process-monitor > thread to stop in such a way that leaves the output monitor threads orphaned. > I've submitted a Github issue and pull-request against the KPL project: > https://github.com/awslabs/amazon-kinesis-producer/issues/93 > https://github.com/awslabs/amazon-kinesis-producer/pull/94 > This issue is rooted in the Amazon Kinesis Producer Library (KPL) that the > Flink Kinesis streaming connector depends upon. It ought to be fixed in the > KPL, but I want to document it on the Flink project. The Flink KPL dependency > should be updated once the KPL has been fixed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (FLINK-5946) Kinesis Producer uses KPL that orphans threads that consume 100% CPU
[ https://issues.apache.org/jira/browse/FLINK-5946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011273#comment-16011273 ] Scott Kidder commented on FLINK-5946: - FYI: the pull-request against the Kinesis Producer Library (KPL) was accepted and merged into the master branch of the KPL project repo. I imagine it'll be present in the next release of the KPL. > Kinesis Producer uses KPL that orphans threads that consume 100% CPU > > > Key: FLINK-5946 > URL: https://issues.apache.org/jira/browse/FLINK-5946 > Project: Flink > Issue Type: Bug > Components: Kinesis Connector >Affects Versions: 1.2.0 >Reporter: Scott Kidder > > It's possible for the Amazon Kinesis Producer Library (KPL) to leave orphaned > threads running after the producer has been instructed to shutdown via the > `destroy()` method. These threads run in a very tight infinite loop that can > push CPU usage to 100%. I've seen this happen on several occasions, though it > does not happen all of the time. Once these threads are orphaned, the only > solution to bring CPU utilization back down is to restart the Flink Task > Manager. > When a KPL producer is instantiated, it creates several threads: one to > execute and monitor the native sender process, and two threads to monitor the > process' stdout and stderr output. It's possible for the process-monitor > thread to stop in such a way that leaves the output monitor threads orphaned. > I've submitted a Github issue and pull-request against the KPL project: > https://github.com/awslabs/amazon-kinesis-producer/issues/93 > https://github.com/awslabs/amazon-kinesis-producer/pull/94 > This issue is rooted in the Amazon Kinesis Producer Library (KPL) that the > Flink Kinesis streaming connector depends upon. It ought to be fixed in the > KPL, but I want to document it on the Flink project. The Flink KPL dependency > should be updated once the KPL has been fixed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
