frankisawesome commented on PR #13608:
URL: https://github.com/apache/trafficserver/pull/13608#issuecomment-6329719753
Hey guys @masaori335 & @bneradt
It appears we've hit this bug in production & could we nudge for a cherry
pick into a new release for this fix potentially
We use ATS as a S3 byte range request cache, for a cloud location that
frequently reads on prem storage.
Below is claude slop, apologies in advance.
```
**Setup.** Stock cache code (our only patches are to the slice and
cache_range_requests plugins). Regular-file spans: 8 × 468 GB on one NVMe drive
per node, slice plugin with 256 KiB blocks, `read_while_writer` enabled.
**Runtime hang.** One stripe's AIO thread sat at 100% user CPU in:
#0 dir_clean_bucket(Dir*, int, StripeSM*)
#1 Directory::cleanup(StripeSM*)
#2 StripeSM::agg_wrap()
#3 StripeSM::aggWrite(int, void*)
#4 StripeSM::aggWriteDone(int, Event*)
#5 AIOCallback::io_complete(int, void*)
#6 AIOThreadInfo::aio_thread_main(AIOThreadInfo*)
It held the stripe mutex the whole time, so every lookup on that stripe
waited:
- `proxy.process.cache.read.active` climbed to about 50k;
- `proxy.process.cache.stripe.lock_contention` reached about 4M/s, against
about 1k/s on healthy nodes;
- the disk sat idle;
- the ET_NET threads used about 17% of a core each, retrying the lock.
The exec_thread watchdog logged nothing, because it doesn't watch ET_AIO
threads.
**Directory state** (walked from gdb):
- The loop is on a bucket head (`p == NULL`) whose offset is 0, in the
head-of-bucket branch of the delete path.
- Following `next` from the head, all 13,536 entries walked had offset 0.
The bucket chain had merged into the segment free list.
- The free list contains a cycle of 8,120 entries, first repeating at step
5,416, so `next == 0` is never reached.
**The corruption persisted on disk.** After a restart, the same stripe hung
in startup recovery, and the watchdog did fire (ET_NET):
#0 dir_clean_bucket(Dir*, int, StripeSM*)
#1 Directory::clear_range(long, long, StripeSM*)
#2 StripeSM::handle_recover_from_data(int, void*)
#3 AIOCallback::io_complete(int, void*)
We recovered only by deleting that span's file.
**When it formed.** The same stripe's previous wrap, about 8.5 h earlier,
completed normally. `Directory::cleanup()` walks every bucket, so the chain was
intact then and the cycle formed during normal running, not at a restart.
**Possible cause.** #13840 says `delete_entry()` moving the next entry into
the head slot "can invalidate another reader's `last_collision` cursor". That
is the head-of-bucket path where our loop sits. A lookup left holding a freed
entry would explain a bucket chain linked into the free list. We haven't proven
it, but if that's right, backporting #13840 alongside this one would fix the
cause as well as the hang.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
