Simon A. Boggis wrote: > I routinely log the output of 'ipfstat -s' (and many other things) to a > file every couple of minutes to help diagnose resource exhaustion. > Running the 'ipfstat -s' output through a small analysis program shows > that the number of 'active' states is permanently hovering dangerously > near or hitting the maximum, hence the increasing 'maximum' counter. > > Just like Wes, we only see a small number of state entries in the table, > which makes it impossible to see with-what and why the table is full: [...] > We've been forced to reboot the machine to clear the problem, but > already I notice that there is a considerable disparity between the > entries in the state table and the 'active' states: [...] > I'm now wondering what the 'invisible' states are (Darren mentioned > 'orphans'), how they get created and if there is a way to clear them and > free up resources again.
Further to the above, I've been doing some analysis of my logs of the output of 'ipfstat -s'. One of the more interesting things is to look at the change in the values of: active bkts in use the difference (active - bkts in use) 0 between successive 'ipfstat -s' runs (every two minutes for me) - I'm calling this 'Delta'. If I total up the running changes (which could be positive or negative) - I'm calling this 'Total Delta', I would expect that, on average they are about zero as states get created, live their lives and die. This doesn't seem to be the case - I plotted all the data I have for the troubled machine (Sep-Dec) and found that bkts in use goes up and down as expected, however there is a constant, and increasing in steps, contribution from the 'hidden' active states in 'active'. This number only gets reset when the router crashes or is rebooted (the former has been happening disturbingly often). I've put the graph here: http://webspace.qmul.ac.uk/saboggis/resources.log.sep-dec_total_delta.png The 'hidden' active states do look like they are never getting cleaned up, and are just consuming resources. Best wishes, Simon -- ---------------------------------------------------------------------- Dr Simon A. Boggis Senior Network Analyst Computing Services, Tel. 020 7882 7078 Queen Mary, University of London, London E1 4NS UK.
