[jira] [Closed] (FLINK-4660) HadoopFileSystem (with S3A) may leak connections, which cause job to stuck in a restarting loop
[ https://issues.apache.org/jira/browse/FLINK-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-4660. --- Resolution: Fixed Release Note: Closing the reopened issue to update versions in which this is fixed > HadoopFileSystem (with S3A) may leak connections, which cause job to stuck in > a restarting loop > --- > > Key: FLINK-4660 > URL: https://issues.apache.org/jira/browse/FLINK-4660 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Reporter: Zhenzhong Xu >Priority: Critical > Fix For: 1.4.0, 1.3.2 > > Attachments: Screen Shot 2016-09-20 at 2.49.14 PM.png, Screen Shot > 2016-09-20 at 2.49.32 PM.png > > > Flink job with checkpoints enabled and configured to use S3A file system > backend, sometimes experiences checkpointing failure due to S3 consistency > issue. This behavior is also reported by other people and documented in > https://issues.apache.org/jira/browse/FLINK-4218. > This problem gets magnified by current HadoopFileSystem implementation, which > can potentially leak S3 client connections, and eventually get into a > restarting loop with “Timeout waiting for a connection from pool” exception > thrown from aws client. > I looked at the code, seems HadoopFileSystem.java never invoke close() method > on fs object upon failure, but the FileSystem may be re-initialized every > time the job gets restarted. > A few evidence I observed: > 1. When I set the connection pool limit to 128, and below commands shows 128 > connections are stuck in CLOSE_WAIT state. > !Screen Shot 2016-09-20 at 2.49.14 PM.png|align=left, vspace=5! > 2. task manager logs indicates that state backend file system consistently > getting initialized upon job restarting. > !Screen Shot 2016-09-20 at 2.49.32 PM.png! > 3. Log indicates there is NPE during cleanning up of stream task which was > caused by “Timeout waiting for connection from pool” exception when trying to > create a directory in S3 bucket. > 2016-09-02 08:17:50,886 ERROR > org.apache.flink.streaming.runtime.tasks.StreamTask - Error during cleanup of > stream task > java.lang.NullPointerException > at > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.cleanup(OneInputStreamTask.java:73) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:323) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:589) > at java.lang.Thread.run(Thread.java:745) > 4.It appears StreamTask from invoking checkpointing operation, to handling > failure, there is no logic associated with closing Hadoop File System object > (which internally includes S3 aws client object), which resides in > HadoopFileSystem.java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (FLINK-4660) HadoopFileSystem (with S3A) may leak connections, which cause job to stuck in a restarting loop
[ https://issues.apache.org/jira/browse/FLINK-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenzhong Xu closed FLINK-4660. --- > HadoopFileSystem (with S3A) may leak connections, which cause job to stuck in > a restarting loop > --- > > Key: FLINK-4660 > URL: https://issues.apache.org/jira/browse/FLINK-4660 > Project: Flink > Issue Type: Bug > Components: State Backends, Checkpointing >Reporter: Zhenzhong Xu >Priority: Critical > Attachments: Screen Shot 2016-09-20 at 2.49.14 PM.png, Screen Shot > 2016-09-20 at 2.49.32 PM.png > > > Flink job with checkpoints enabled and configured to use S3A file system > backend, sometimes experiences checkpointing failure due to S3 consistency > issue. This behavior is also reported by other people and documented in > https://issues.apache.org/jira/browse/FLINK-4218. > This problem gets magnified by current HadoopFileSystem implementation, which > can potentially leak S3 client connections, and eventually get into a > restarting loop with “Timeout waiting for a connection from pool” exception > thrown from aws client. > I looked at the code, seems HadoopFileSystem.java never invoke close() method > on fs object upon failure, but the FileSystem may be re-initialized every > time the job gets restarted. > A few evidence I observed: > 1. When I set the connection pool limit to 128, and below commands shows 128 > connections are stuck in CLOSE_WAIT state. > !Screen Shot 2016-09-20 at 2.49.14 PM.png|align=left, vspace=5! > 2. task manager logs indicates that state backend file system consistently > getting initialized upon job restarting. > !Screen Shot 2016-09-20 at 2.49.32 PM.png! > 3. Log indicates there is NPE during cleanning up of stream task which was > caused by “Timeout waiting for connection from pool” exception when trying to > create a directory in S3 bucket. > 2016-09-02 08:17:50,886 ERROR > org.apache.flink.streaming.runtime.tasks.StreamTask - Error during cleanup of > stream task > java.lang.NullPointerException > at > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.cleanup(OneInputStreamTask.java:73) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:323) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:589) > at java.lang.Thread.run(Thread.java:745) > 4.It appears StreamTask from invoking checkpointing operation, to handling > failure, there is no logic associated with closing Hadoop File System object > (which internally includes S3 aws client object), which resides in > HadoopFileSystem.java. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
