taoyunxing created TS-4279:
------------------------------
Summary: ats fallen into dead loop for cache directory overflow
Key: TS-4279
URL: https://issues.apache.org/jira/browse/TS-4279
Project: Traffic Server
Issue Type: Bug
Components: Cache
Reporter: taoyunxing
CPU 40 cores, Mem: 120GB, Disk: 1*300 sys + 11 * 899GB,
records.config:
CONFIG proxy.config.cache.min_average_object_size INT 1048576
CONFIG proxy.config.cache.ram_cache.algorithm INT 1
CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304
CONFIG proxy.config.cache.ram_cache.size INT 64424509440
storage.config:
/dev/sdc id=cache.disk.1
I encountered a kind of dead loop situation of ats 5.3.1 on two production
hosts, a burst of warning is seen by me in the diags.log like this:
{code}
[Mar 16 13:04:32.730] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
[Mar 16 13:04:32.732] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
[Mar 16 13:04:32.733] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
[Mar 16 13:04:32.735] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
[Mar 16 13:04:32.737] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
[Mar 16 13:04:32.739] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
[Mar 16 13:04:32.742] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
[Mar 16 13:04:32.744] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
[Mar 16 13:04:32.747] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
[Mar 16 13:04:32.750] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
[Mar 16 13:04:32.753] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
[Mar 16 13:04:32.756] Server {0x2b8ffc544700} WARNING: <CacheDir.cc:502
(freelist_clean)> cache directory overflow on '/dev/sde' segment 4, purging...
{code}
ats restart in every serval hours, and the TIMEWAIT count is huge above the
ESTABLISH TCP connection count.
the following is the current dir snapshot of those host:
{code}
Directory for [cache.disk.1 172032:109741163]
Bytes: 8573600
Segments: 14
Buckets: 15310
Entries: 857360
Full: 852904
Empty: 4085
Stale: 0
Free: 371
Bucket Fullness: 4085 15800 32044 41621
42175 33137 22232 12605
Segment Fullness: 60903 60918 60914 60947 60956
60947 60872 60943 60918 60927
60858 60917 60927 60957
Freelist Fullness: 45 30 27 13 0
7 89 5 32 12
83 0 20 8
{code}
I wonder why, anyone help me? thinks a lot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
