[
https://issues.apache.org/jira/browse/TS-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258900#comment-14258900
]
Sudheer Vinukonda edited comment on TS-3261 at 12/26/14 6:44 AM:
-----------------------------------------------------------------
Running v5.2 with valgrind, it seems that this leak may also be related to the
fixes in TS-3006.
{code}
==5849== 422,520 (348,096 direct, 74,424 indirect) bytes in 3,108 blocks are
definitely lost in loss record 3,811 of 3,891
==5849== at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==5849== by 0x3C9606ABED: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.1e)
==5849== by 0x3C960DC506: BIO_new (in /usr/lib64/libcrypto.so.1.0.1e)
==5849== by 0x3C960DCAE1: BIO_new_mem_buf (in /usr/lib64/libcrypto.so.1.0.1e)
==5849== by 0x73D543: SSLNetVConnection::read_raw_data()
(SSLNetVConnection.cc:367)
==5849== by 0x73F5AF: SSLNetVConnection::sslServerHandShakeEvent(int&)
(SSLNetVConnection.cc:950)
==5849== by 0x73F2BA: SSLNetVConnection::sslStartHandShake(int, int&)
(SSLNetVConnection.cc:874)
==5849== by 0x73D85F: SSLNetVConnection::net_read_io(NetHandler*, EThread*)
(SSLNetVConnection.cc:433)
==5849== by 0x74C488: NetHandler::mainNetEvent(int, Event*) (UnixNet.cc:399)
==5849== by 0x502EAD: Continuation::handleEvent(int, void*)
(I_Continuation.h:146)
==5849== by 0x7730C1: EThread::process_event(Event*, int)
(UnixEThread.cc:144)
==5849== by 0x7735CB: EThread::execute() (UnixEThread.cc:268)
{code}
{code}
==5849== 1,074,816 (885,360 direct, 189,456 indirect) bytes in 7,905 blocks are
definitely lost in loss record 3,841 of 3,891
==5849== at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==5849== by 0x3C9606ABED: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.1e)
==5849== by 0x3C960DC506: BIO_new (in /usr/lib64/libcrypto.so.1.0.1e)
==5849== by 0x73C9E9: make_ssl_connection(ssl_ctx_st*, SSLNetVConnection*)
(SSLNetVConnection.cc:143)
==5849== by 0x73F1FD: SSLNetVConnection::sslStartHandShake(int, int&)
(SSLNetVConnection.cc:866)
==5849== by 0x73D85F: SSLNetVConnection::net_read_io(NetHandler*, EThread*)
(SSLNetVConnection.cc:433)
==5849== by 0x74C488: NetHandler::mainNetEvent(int, Event*) (UnixNet.cc:399)
==5849== by 0x502EAD: Continuation::handleEvent(int, void*)
(I_Continuation.h:146)
==5849== by 0x7730C1: EThread::process_event(Event*, int)
(UnixEThread.cc:144)
==5849== by 0x7735CB: EThread::execute() (UnixEThread.cc:268)
==5849== by 0x77267C: spawn_thread_internal(void*) (Thread.cc:88)
==5849== by 0x52ED850: start_thread (in /lib64/libpthread-2.12.so)
{code}
[~shinrich]: I see that {{SSLNetVConnection::read_raw_data()}} calls
{{BIO_new_mem_buf}} to create a new memory bio and sets it as the SSL's BIO,
but, I don't see any {{BIO_free}} of the old bio. Also, similar code in
{{make_ssl_connection}}. Doesn't that result in a leak?
was (Author: sudheerv):
Running v5.2 with valgrind, it seems that this leak may also be related to the
fixes in TS-3006.
{code}
==5849== 422,520 (348,096 direct, 74,424 indirect) bytes in 3,108 blocks are
definitely lost in loss record 3,811 of 3,891
==5849== at 0x4A069EE: malloc (vg_replace_malloc.c:270)
==5849== by 0x3C9606ABED: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.1e)
==5849== by 0x3C960DC506: BIO_new (in /usr/lib64/libcrypto.so.1.0.1e)
==5849== by 0x3C960DCAE1: BIO_new_mem_buf (in /usr/lib64/libcrypto.so.1.0.1e)
==5849== by 0x73D543: SSLNetVConnection::read_raw_data()
(SSLNetVConnection.cc:367)
==5849== by 0x73F5AF: SSLNetVConnection::sslServerHandShakeEvent(int&)
(SSLNetVConnection.cc:950)
==5849== by 0x73F2BA: SSLNetVConnection::sslStartHandShake(int, int&)
(SSLNetVConnection.cc:874)
==5849== by 0x73D85F: SSLNetVConnection::net_read_io(NetHandler*, EThread*)
(SSLNetVConnection.cc:433)
==5849== by 0x74C488: NetHandler::mainNetEvent(int, Event*) (UnixNet.cc:399)
==5849== by 0x502EAD: Continuation::handleEvent(int, void*)
(I_Continuation.h:146)
==5849== by 0x7730C1: EThread::process_event(Event*, int)
(UnixEThread.cc:144)
==5849== by 0x7735CB: EThread::execute() (UnixEThread.cc:268)
{code}
[~shinrich]: I see that {{SSLNetVConnection::read_raw_data()}} calls
{{BIO_new_mem_buf}} to create a new memory bio, but, I don't see any
corresponding {{BIO_free}} to release the bio. Doesn't that result in a leak?
> possible slow leak in v5.2.0
> ----------------------------
>
> Key: TS-3261
> URL: https://issues.apache.org/jira/browse/TS-3261
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.2.0
> Reporter: Sudheer Vinukonda
> Assignee: Sudheer Vinukonda
> Priority: Critical
> Fix For: 5.3.0
>
>
> After fixing the iobuffer leak in TS-3257, the iobuffers seem stable on
> v5.2.0, but, there still seems to be a slow memory leak. The RES memory from
> top shows 15g after running v5.2.0 in prod for more than 3 days, whereas the
> corresponding v5.0 host shows 10g after running for more than a week.
> Below is the dump of iobuffers between the v5.2.0 and v5.0 host - as
> expected, most iobufs are lower than v5.0 host (since, the v5.0 host been
> running longer), except the 32k buffer (iobuf[8]). But, the leak doesn't seem
> to be explained by the difference in 32k buffers either, as it is not high
> enough to explain the 5g difference in total memory.
> v5.2.0 host:
> {code}
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 67108864 | 25165824 | 2097152 |
> memory/ioBufAllocator[14]
> 2013265920 | 1825570816 | 1048576 |
> memory/ioBufAllocator[13]
> 620756992 | 549978112 | 524288 |
> memory/ioBufAllocator[12]
> 780140544 | 593494016 | 262144 |
> memory/ioBufAllocator[11]
> 742391808 | 574619648 | 131072 |
> memory/ioBufAllocator[10]
> 901775360 | 735576064 | 65536 |
> memory/ioBufAllocator[9]
> 1189085184 | 1093304320 | 32768 |
> memory/ioBufAllocator[8]
> 474480640 | 348733440 | 16384 |
> memory/ioBufAllocator[7]
> 269221888 | 211320832 | 8192 |
> memory/ioBufAllocator[6]
> 156762112 | 142999552 | 4096 |
> memory/ioBufAllocator[5]
> 0 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 131072 | 0 | 1024 |
> memory/ioBufAllocator[3]
> 65536 | 0 | 512 |
> memory/ioBufAllocator[2]
> 65536 | 256 | 256 |
> memory/ioBufAllocator[1]
> 16384 | 0 | 128 |
> memory/ioBufAllocator[0]
> {code}
> v.5.0.0 host:
> {code}
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 134217728 | 31457280 | 2097152 |
> memory/ioBufAllocator[14]
> 2147483648 | 1843396608 | 1048576 |
> memory/ioBufAllocator[13]
> 788529152 | 608174080 | 524288 |
> memory/ioBufAllocator[12]
> 897581056 | 680525824 | 262144 |
> memory/ioBufAllocator[11]
> 796917760 | 660471808 | 131072 |
> memory/ioBufAllocator[10]
> 985661440 | 818479104 | 65536 |
> memory/ioBufAllocator[9]
> 873463808 | 677969920 | 32768 |
> memory/ioBufAllocator[8]
> 544735232 | 404439040 | 16384 |
> memory/ioBufAllocator[7]
> 310902784 | 237887488 | 8192 |
> memory/ioBufAllocator[6]
> 160956416 | 115515392 | 4096 |
> memory/ioBufAllocator[5]
> 0 | 0 | 2048 |
> memory/ioBufAllocator[4]
> 131072 | 2048 | 1024 |
> memory/ioBufAllocator[3]
> 65536 | 0 | 512 |
> memory/ioBufAllocator[2]
> 98304 | 50688 | 256 |
> memory/ioBufAllocator[1]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
