David Carlin created TS-1316:
--------------------------------
Summary: ATS connection refused once cache direntries exhausted
Key: TS-1316
URL: https://issues.apache.org/jira/browse/TS-1316
Project: Traffic Server
Issue Type: Bug
Components: Cache
Affects Versions: 3.2.0
Environment: RHEL 6.2 x86_64
Reporter: David Carlin
We had a pair of ATS 3.2.0 boxes that stopped passing traffic simultaneously.
Here are the traffic.out msgs we saw on both boxes:
[Jun 22 05:53:27.637] Server {0x2b6b6d9da700} WARNING: cache directory overflow
on '/dev/dm-4' segment 36, purging...
[Jun 22 05:56:05.542] Server {0x2b6b6d2d3700} WARNING: cache directory overflow
on '/dev/dm-4' segment 85, purging...
[Jun 22 05:56:07.434] Server {0x2b6b6d4d5700} WARNING: cache directory overflow
on '/dev/dm-4' segment 71, purging...
[Jun 22 05:58:24.743] Server {0x2b6b6d8d9700} WARNING: cache directory overflow
on '/dev/dm-4' segment 33, purging...
Those messages went on for a couple minutes, then traffic apparently ceased -
our monitoring system saw connection refused for port 80 on ATS from then on.
The connection refused state went on for many hours until ATS was restarted.
There were no traffic_cop msgs in /var/log/messages indicating that the
heartbeat failed.
Here are the relevant ATS settings/stats:
proxy.process.cache.bytes_total = 190690320384
proxy.process.cache.direntries.total = 5817752
proxy.config.cache.min_average_object_size = 32768
We previously came up with proxy.config.cache.min_average_object_size by
waiting for the cache to fill and dividing proxy.process.cache.bytes_used by
proxy.process.cache.direntries.used - which equals about 34KB.
We're assuming ATS ran out of direntries and it didn't handle this situation
gracefully. As a workaround, I'm going to lower
proxy.process.cache.direntries.used to 24KB.
Thanks to Bryan Call for helping me troubleshoot this!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
