[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory

2017-03-13 Thread bryancall
Github user bryancall commented on the issue: https://github.com/apache/trafficserver/issues/1532 Information from 7.0.0 release running in production on the same server: ``` [bcall@e24 crash]$ ss -tn | awk '{print $4}' | grep -c ':80$' ; ss -tn | awk '{print $4}' | grep -c

[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory

2017-03-13 Thread bryancall
Github user bryancall commented on the issue: https://github.com/apache/trafficserver/issues/1532 The problem I am seeing with 7.1 has to do with http2 client sessions not closing and releasing the `MIOBuffer`. I am running a test with 7.0.0 and seeing if the issue was there too.

[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory

2017-03-13 Thread mingzym
Github user mingzym commented on the issue: https://github.com/apache/trafficserver/issues/1532 I'd suggest you should take a look what @scw00 commented, base on man malloc: ``` Normally, malloc() allocates memory from the heap, and adjusts the size of the heap as required,

[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory

2017-03-13 Thread shinrich
Github user shinrich commented on the issue: https://github.com/apache/trafficserver/issues/1532 Built with jemalloc and installed on same machine. Does not crash anymore, but the load average is crazy (25-30). spinlock shows up as 25% CPU in perf top and IOBuffer::read_avail shows

[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory

2017-03-10 Thread shinrich
Github user shinrich commented on the issue: https://github.com/apache/trafficserver/issues/1532 I installed on a machine with a higher load, and it fails in ats_memalign from freelist_new within 2 minutes of traffic. Watching top the amount of memory used by the process is much

[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory

2017-03-08 Thread bryancall
Github user bryancall commented on the issue: https://github.com/apache/trafficserver/issues/1532 There is an issue with the number of real TCP connections, the stats that keeps track of the number of current client connections, and the number of http2 client sessions. All of them

[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory

2017-03-06 Thread scw00
Github user scw00 commented on the issue: https://github.com/apache/trafficserver/issues/1532 We have meet the same situation in our product env, and we try to modified max_map_count, the kernel variable, and it work well. In Short, it may too much map nnon in process, and

[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory

2017-03-06 Thread shinrich
Github user shinrich commented on the issue: https://github.com/apache/trafficserver/issues/1532 I saw stacks like this when I was running with the free list disabled (-f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory

2017-03-03 Thread shinrich
Github user shinrich commented on the issue: https://github.com/apache/trafficserver/issues/1532 Any indication of object buildup in a particular type of object pool? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] trafficserver issue #1532: ATS 7.1 release running out of memory

2017-03-03 Thread bryancall
GitHub user bryancall opened an issue: https://github.com/apache/trafficserver/issues/1532 ATS 7.1 release running out of memory Here is a backtrace from a bunch of cores I am getting from 7.1. They are mostly issues with memory allocation: [bcall@e24 ~]$ egrep '\#2 |\#3 |\#4