Josh, When you see this happen again, please try `lsof -n -P -p <pid>` (note the -n and -P) instead. That will give the IP addrs and port numbers without attempting to convert host or service names and should help you identify the offending connections.
Regards, -- Mike Rylander | President | Equinox Software, Inc. / The Open Source Experts | phone: 1-877-OPEN-ILS (673-6457) | email: [email protected] | web: http://www.esilibrary.com On Wed, Jul 22, 2015 at 9:14 PM, Josh Stompro <[email protected]> wrote: > Greetings, I’ve been trying to figure out why my two front end > Evergreen application servers keep hitting some resource limits having to > do with tcp sockets (numtcpsock openvz beancounters). > > > > I’m running EG 2.8.2, OpenSRF 2.4.1, Debian Jessie in an Openvz container > on Proxmox VE 3.4 > > > > Nothing looks out of the ordinary when I look at the output of ‘ss –s’ or > ‘netstat –a’, but the numtcpsock counter keeps going up, until I have 5000+ > reported open tcp socket connections. > > > > I think I’ve narrowed it down to apache, since restarting apache resets > the numtcpsock numbers back in line with what is reported by ‘ss –s’ > > > > If I take a look at all the open fd’s of an apache process, I see a bunch > of the following. So I think some socket connections are being opened but > not closed properly. > > > > (lsof –p <pid>) > > > > /usr/sbin 11821 opensrf 171u sock 0,6 0t0 61135031 > can't identify protocol > > /usr/sbin 11821 opensrf 172u sock 0,6 0t0 61135034 > can't identify protocol > > /usr/sbin 11821 opensrf 173u sock 0,6 0t0 61135037 > can't identify protocol > > /usr/sbin 11821 opensrf 174u sock 0,6 0t0 61321969 > can't identify protocol > > /usr/sbin 11821 opensrf 175u sock 0,6 0t0 61321972 > can't identify protocol > > /usr/sbin 11821 opensrf 176u sock 0,6 0t0 61321975 > can't identify protocol > > /usr/sbin 11821 opensrf 177u sock 0,6 0t0 61321978 > can't identify protocol > > /usr/sbin 11821 opensrf 178u sock 0,6 0t0 61321981 > can't identify protocol > > /usr/sbin 11821 opensrf 179u sock 0,6 0t0 61458539 > can't identify protocol > > /usr/sbin 11821 opensrf 180u sock 0,6 0t0 61458542 > can't identify protocol > > /usr/sbin 11821 opensrf 181u sock 0,6 0t0 61458545 > can't identify protocol > > /usr/sbin 11821 opensrf 182u sock 0,6 0t0 61458548 > can't identify protocol > > /usr/sbin 11821 opensrf 183u sock 0,6 0t0 61458551 > can't identify protocol > > /usr/sbin 11821 opensrf 184u sock 0,6 0t0 62085495 > can't identify protocol > > /usr/sbin 11821 opensrf 185u sock 0,6 0t0 62085498 > can't identify protocol > > /usr/sbin 11821 opensrf 186u sock 0,6 0t0 62085501 > can't identify protocol > > /usr/sbin 11821 opensrf 187u sock 0,6 0t0 62085504 > can't identify protocol > > /usr/sbin 11821 opensrf 188u sock 0,6 0t0 62085507 > can't identify protocol > > /usr/sbin 11821 opensrf 189u sock 0,6 0t0 63801157 > can't identify protocol > > /usr/sbin 11821 opensrf 190u sock 0,6 0t0 63801160 > can't identify protocol > > /usr/sbin 11821 opensrf 191u sock 0,6 0t0 63801163 > can't identify protocol > > /usr/sbin 11821 opensrf 192u sock 0,6 0t0 63801166 > can't identify protocol > > /usr/sbin 11821 opensrf 193u sock 0,6 0t0 63801169 > can't identify protocol > > /usr/sbin 11821 opensrf 194u sock 0,6 0t0 63961716 > can't identify protocol > > /usr/sbin 11821 opensrf 195u sock 0,6 0t0 63961719 > can't identify protocol > > /usr/sbin 11821 opensrf 196u sock 0,6 0t0 63961722 > can't identify protocol > > /usr/sbin 11821 opensrf 197u sock 0,6 0t0 63961725 > can't identify protocol > > /usr/sbin 11821 opensrf 198u sock 0,6 0t0 63961728 > can't identify protocol > > /usr/sbin 11821 opensrf 199u sock 0,6 0t0 64808966 > can't identify protocol > > /usr/sbin 11821 opensrf 200u sock 0,6 0t0 64808971 > can't identify protocol > > /usr/sbin 11821 opensrf 201u sock 0,6 0t0 64808974 > can't identify protocol > > /usr/sbin 11821 opensrf 202u sock 0,6 0t0 64808977 > can't identify protocol > > /usr/sbin 11821 opensrf 203u sock 0,6 0t0 64808980 > can't identify protocol > > > > I’m not sure how to track down the problem, I’ll try using strace to see > what connections are being created, but I’m not quite sure what to look for. > > > > If anyone has run into this before, please let me know. > > Josh >
