Re: jobmanager holds too many CLOSE_WAIT connection to datanode

[vino yang]

Hi Youjun,

How long has your job been running for a long time?
As far as I know, if in a short time, for checkpoint, jobmanager will not
generate so many connections to HDFS.
What is your Flink cluster environment? Standalone or Flink on YARN?
In addition, does JM's log show any timeout information? Has Checkpoint
timed out?
If you can provide more information, it will help locate the problem.

Thanks, vino.

Yuan,Youjun <yuanyou...@baidu.com> 于2018年8月23日周四 下午10:53写道：

> Hi,
>
>
>
> After running for a while , my job manager holds thousands of CLOSE_WAIT
> TCP connection to HDFS datanode, the number is growing up slowly, and it’s
> likely will hit the max open file limit. My jobs checkpoint to HDFS every
> minute.
>
> If I run lsof -i -a -p $JMPID, I can get a tons of following output:
>
> java    9433  iot  408u  IPv4 4060901898      0t0  TCP
> jmHost:17922->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  409u  IPv4 4061478455      0t0  TCP
> jmHost:52854->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  410r  IPv4 4063170767      0t0  TCP
> jmHost:49384->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  411w  IPv4 4063188376      0t0  TCP
> jmHost:50516->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  412u  IPv4 4061459881      0t0  TCP
> jmHost:51651->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  413u  IPv4 4063737603      0t0  TCP
> jmHost:31318->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  414w  IPv4 4062030625      0t0  TCP
> jmHost:34033->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  415u  IPv4 4062049134      0t0  TCP
> jmHost:35156->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  416u  IPv4 4062615550      0t0  TCP
> jmHost:16962->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  417r  IPv4 4063757056      0t0  TCP
> jmHost:32553->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  418w  IPv4 4064304789      0t0  TCP
> jmHost:13375->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  419u  IPv4 4062599328      0t0  TCP
> jmHost:15915->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  420w  IPv4 4065462963      0t0  TCP
> jmHost:30432->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  421u  IPv4 4067178257      0t0  TCP
> jmHost:28334->datanode:50010 (CLOSE_WAIT)
>
> java    9433  iot  422u  IPv4 4066022066      0t0  TCP
> jmHost:11843->datanode:50010 (CLOSE_WAIT)
>
>
>
>
>
> I know restarting the job manager should cleanup those connections, but I
> wonder if there is any better solution?
>
> Btw, I am using flink 1.4.0, and running a standalone cluster.
>
>
>
> Thanks
>
> Youjun
>