Colleagues of mine have communicated that this has been seen in the past due to interaction between the Spectrum Scale performance monitor (zimon) and Grafana. Are you using Grafana? Normally zimon is configured to use local port 9094 so if that is the port which the CLOSE_WAIT is attached then it would seem to be an instance of this problem. You can use the following to check for this condition.
netstat -ntp | grep "\:9094 .*CLOSE_WAIT" | wc -l
Fred
__________________________________________________
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
[email protected]
Fred Stock | IBM Pittsburgh Lab | 720-430-8821
[email protected]
----- Original message -----
From: Leonardo Sala <[email protected]>
Sent by: [email protected]
To: "[email protected]" <[email protected]>
Cc:
Subject: [EXTERNAL] [gpfsug-discuss] Thousands of CLOSE_WAIT IPV6 connections on CES
Date: Fri, Feb 21, 2020 9:30 AM
Dear all,
I was wondering if anybody recently encountered a similar issue (I found a related thread from 2018, but it was inconclusive). I just found that one of our production CES nodes have 28k CLOSE_WAIT tcp6 connections, I do not understand why... the second node in the same cluster does not have this issue. Both are:
- GPFS 5.0.4.2
- RHEL 7.4
has anybody else encountered anything similar? In the last few days it seems it happened once on one node, and twice on the other, but never on both...
Thanks for any feedback!
cheers
leo
--
Paul Scherrer Institut
Dr. Leonardo Sala
Group Leader High Performance Computing
Deputy Section Head Science IT
Science IT
WHGA/036
Forschungstrasse 111
5232 Villigen PSI
Switzerland
Phone: +41 56 310 3369
[email protected]
www.psi.ch_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
