[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory
Github user bryancall commented on the issue:
https://github.com/apache/trafficserver/issues/1532
Information from 7.0.0 release running in production on the same server:
```
[bcall@e24 crash]$ ss -tn | awk '{print $4}' | grep -c ':80$' ; ss -tn |
awk '{print $4}' | grep -c ':443$'; traffic_ctl metric get
proxy.process.net.connections_currently_open; traffic_ctl metric match
current_client_sessions; traffic_ctl metric match
http.current_client_connections; grep HttpSM.cc:6485
/home/y/logs/trafficserver/traffic.out; ss -s
2658
54005
proxy.process.net.connections_currently_open 78128
proxy.process.http2.current_client_sessions 37296
proxy.process.http.current_client_connections 14835
7748180 |7669410 | 2563600384 | 32545 |
memory/IOBuffer/../../../../trafficserver/proxy/http/HttpSM.cc:6485
27926828 | 27884703 | 1371799552 | 32564 |
memory/IOBuffer/../../../../trafficserver/proxy/http/HttpSM.cc:6485
30031206 | 2975 | 1009647616 | 32328 |
memory/IOBuffer/../../../../trafficserver/proxy/http/HttpSM.cc:6485
33306877 | 33252469 | 1767923712 | 32493 |
memory/IOBuffer/../../../../trafficserver/proxy/http/HttpSM.cc:6485
87611972 | 87588769 |752799744 | 32444 |
memory/IOBuffer/../../../../trafficserver/proxy/http/HttpSM.cc:6485
Total: 63568 (kernel 65390)
TCP: 97209 (estab 63217, closed 31012, orphaned 2935, synrecv 0, timewait
31012/0), ports 17009
Transport Total IPIPv6
* 65390 - -
RAW 0 0 0
UDP 281216
TCP 66197 50229 15968
INET 66225 50241 15984
FRAG 0 0 0
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory
Github user bryancall commented on the issue: https://github.com/apache/trafficserver/issues/1532 The problem I am seeing with 7.1 has to do with http2 client sessions not closing and releasing the `MIOBuffer`. I am running a test with 7.0.0 and seeing if the issue was there too. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory
Github user mingzym commented on the issue: https://github.com/apache/trafficserver/issues/1532 I'd suggest you should take a look what @scw00 commented, base on man malloc: ``` Normally, malloc() allocates memory from the heap, and adjusts the size of the heap as required, using sbrk(2). When allocating blocks of memory larger than MMAP_THRESHOLD bytes, the glibc malloc() implementation allocates the memory as a private anonymous mapping using mmap(2). MMAP_THRESHOLD is 128 kB by default, but is adjustable using mallopt(3). Allocations performed using mmap(2) are unaffected by the RLIMIT_DATA resource limit (see getrlimit(2)). ``` a malloc may tigger a mmap() when doing the real alloc, so things may get complex with default sysctrl config, please take a look. and I thinks that before TS-1822 the default M_MMAP_MAX = 2097152, no harm and it will bump the default 64K to 2M, it is a good if you used many mmap in your system. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory
Github user shinrich commented on the issue: https://github.com/apache/trafficserver/issues/1532 Built with jemalloc and installed on same machine. Does not crash anymore, but the load average is crazy (25-30). spinlock shows up as 25% CPU in perf top and IOBuffer::read_avail shows up in second spot at 7-12% cpu. I did notice that as part of fix for TS-1822, @PSUdaemon removes a call to mallopt. Perhaps we still need to set a constraint for the straight glibc malloc? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory
Github user shinrich commented on the issue: https://github.com/apache/trafficserver/issues/1532 I installed on a machine with a higher load, and it fails in ats_memalign from freelist_new within 2 minutes of traffic. Watching top the amount of memory used by the process is much less than the amount of available memory (10% in use). Could be that top is delayed in getting updated. The same build (or very similar build) has run for days on a machine with lower (1/3) of the traffic load. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory
Github user bryancall commented on the issue:
https://github.com/apache/trafficserver/issues/1532
There is an issue with the number of real TCP connections, the stats that
keeps track of the number of current client connections, and the number of
http2 client sessions. All of them disagree:
```
[bcall@e24 ~]$ ss -tn | awk '{print $4}' | grep -c :80 ; ss -tn | awk
'{print $4}' | grep -c :443; traffic_ctl metric get
proxy.process.net.accepts_currently_open; traffic_ctl metric match
current_client_sessions
253
1364
proxy.process.net.accepts_currently_open 9
proxy.process.http2.current_client_sessions 41804
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory
Github user scw00 commented on the issue: https://github.com/apache/trafficserver/issues/1532 We have meet the same situation in our product env, and we try to modified max_map_count, the kernel variable, and it work well. In Short, it may too much map nnon in process, and linux limited it by max_map_count ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory
Github user shinrich commented on the issue: https://github.com/apache/trafficserver/issues/1532 I saw stacks like this when I was running with the free list disabled (-f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory
Github user shinrich commented on the issue: https://github.com/apache/trafficserver/issues/1532 Any indication of object buildup in a particular type of object pool? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory
GitHub user bryancall opened an issue: https://github.com/apache/trafficserver/issues/1532 ATS 7.1 release running out of memory Here is a backtrace from a bunch of cores I am getting from 7.1. They are mostly issues with memory allocation: [bcall@e24 ~]$ egrep '\#2 |\#3 |\#4 |\#5 ' ~/gdb.out > #2 0x007857e8 in UnixNetProcessor::allocate_vc (this=0x105cc20, t=0x0) at ../../../../trafficserver/iocore/net/UnixNetProcessor.cc:460 > #3 0x00782bdf in NetAccept::do_blocking_accept (this=0x2b9b280253d0, t=0x2b9b281d6050) at ../../../../trafficserver/iocore/net/UnixNetAccept.cc:259 > #4 0x00783a1b in NetAccept::acceptLoopEvent (this=0x2b9b280253d0, event=1, e=0x2b9b281cf3c0) at ../../../../trafficserver/iocore/net/UnixNetAccept.cc:469 > #5 0x00515468 in Continuation::handleEvent (this=0x2b9b280253d0, event=1, data=0x2b9b281cf3c0) at /home/bcall/dev/yahoo/build/_build/ats_build/../../trafficserver/iocore/eventsystem/I_Continuation.h:153 > #2 0x2af5d64098d0 in ink_abort (message_format=0x2af5d64200f0 "couldn't allocate %zu bytes at alignment %zu - insufficient memory") at ../../../../trafficserver/lib/ts/ink_error.cc:99 > #3 0x2af5d640deff in ats_memalign (alignment=4096, size=4194304) at ../../../../trafficserver/lib/ts/ink_memory.cc:108 > #4 0x00515638 in IOBufferData::alloc (this=0x2af631221260, size_index=-4194304, type=MEMALIGNED) at /home/bcall/dev/yahoo/build/_build/ats_build/../../trafficserver/iocore/eventsystem/P_IOBuffer.h:288 > #5 0x00515579 in new_IOBufferData_internal (loc=0x814bf0 "memory/IOBuffer/../../../../trafficserver/iocore/cache/Cache.cc:2498", size_index=-4194304, type=MEMALIGNED) at /home/bcall/dev/yahoo/build/_build/ats_build/../../trafficserver/iocore/eventsystem/P_IOBuffer.h:264 > #2 0x2b46476178d0 in ink_abort (message_format=0x2b464762e0f0 "couldn't allocate %zu bytes at alignment %zu - insufficient memory") at ../../../../trafficserver/lib/ts/ink_error.cc:99 > #3 0x2b464761beff in ats_memalign (alignment=4096, size=33554432) at ../../../../trafficserver/lib/ts/ink_memory.cc:108 > #4 0x2b464761c938 in freelist_new (f=0x14ec000) at ../../../../trafficserver/lib/ts/ink_queue.cc:212 > #5 0x2b464761c832 in ink_freelist_new (f=0x14ec000) at ../../../../trafficserver/lib/ts/ink_queue.cc:183 > #2 0x2aed89fc08d0 in ink_abort (message_format=0x2aed89fd70f0 "couldn't allocate %zu bytes at alignment %zu - insufficient memory") at ../../../../trafficserver/lib/ts/ink_error.cc:99 > #3 0x2aed89fc4eff in ats_memalign (alignment=4096, size=987136) at ../../../../trafficserver/lib/ts/ink_memory.cc:108 > #4 0x2aed89fc5938 in freelist_new (f=0x25bb5d0) at ../../../../trafficserver/lib/ts/ink_queue.cc:212 > #5 0x2aed89fc5832 in ink_freelist_new (f=0x25bb5d0) at ../../../../trafficserver/lib/ts/ink_queue.cc:183 > #2 0x2b3d66d4a8d0 in ink_abort (message_format=0x2b3d66d610f0 "couldn't allocate %zu bytes at alignment %zu - insufficient memory") at ../../../../trafficserver/lib/ts/ink_error.cc:99 > #3 0x2b3d66d4eeff in ats_memalign (alignment=4096, size=33554432) at ../../../../trafficserver/lib/ts/ink_memory.cc:108 > #4 0x2b3d66d4f938 in freelist_new (f=0x1f5) at ../../../../trafficserver/lib/ts/ink_queue.cc:212 > #5 0x2b3d66d4f832 in ink_freelist_new (f=0x1f5) at ../../../../trafficserver/lib/ts/ink_queue.cc:183 > #2 0x2af391be38d0 in ink_abort (message_format=0x2af391bfa0f0 "couldn't allocate %zu bytes at alignment %zu - insufficient memory") at ../../../../trafficserver/lib/ts/ink_error.cc:99 > #3 0x2af391be7eff in ats_memalign (alignment=4096, size=3407872) at ../../../../trafficserver/lib/ts/ink_memory.cc:108 > #4 0x00515638 in IOBufferData::alloc (this=0x2ab464602e90, size_index=-3407872, type=MEMALIGNED) at /home/bcall/dev/yahoo/build/_build/ats_build/../../trafficserver/iocore/eventsystem/P_IOBuffer.h:288 > #5 0x00515579 in new_IOBufferData_internal (loc=0x814bf0 "memory/IOBuffer/../../../../trafficserver/iocore/cache/Cache.cc:2498", size_index=-3407872, type=MEMALIGNED) at /home/bcall/dev/yahoo/build/_build/ats_build/../../trafficserver/iocore/eventsystem/P_IOBuffer.h:264 > #2 0x00534d3d in TSHttpSsnClientAddrGet (ssnp=0x2ab21f4d1bc0) at ../../../trafficserver/proxy/InkAPI.cc:5445 > #3 0x00534d8b in TSHttpTxnClientAddrGet (txnp=0x2ab48b84cc00) at ../../../trafficserver/proxy/InkAPI.cc:5453 > #4 0x2aaab248f8f6 in http_hook (contp=0x275cf40, event=60006, edata=0x2ab48b84cc00) at INKPluginInit.cc:174 > #5 0x0052a025 in INKContInternal::handle_event (this=0x275cf40, event=60006, edata=0x2ab48b84cc00) at ../../../trafficserver/proxy/InkAPI.cc:1048 > #2 0x2ab0ec4f18d0 in ink_abort (message_format=0x2ab0ec5074f0 "%s:%d: failed
