[jira] [Comment Edited] (TS-3285) Seg fault when 100 CONT handling is enabled
[
https://issues.apache.org/jira/browse/TS-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271543#comment-14271543
]
Sudheer Vinukonda edited comment on TS-3285 at 1/23/15 4:59 PM:
Further debugging showed that 100 cont implementation calls {{do_io_write()}}
with a reader on MIOBuffer {{ua_entry-write_buffer}}. The point where this
buffer {{ua_entry-write_buffer}} is freed seems to be
{{VC_EVENT_READ_COMPLETE/HTTP_TUNNEL_EVENT_PRECOMPLETE}} for the POST data being
read from the client, instead of a WRITE_COMPLETE event for the 100 cont's
{{do_io_write()}} operation. This could result in premature free'ing of the
buffer, while the WRITE is not complete yet.
Note how do_io_write/write_to_net (that still has a reference to the _writer of
the 100 cont's MIOBuffer), internally may end up allocating iobuf via
write_avail(). This piece of code (which could get executed after the 100 cont
buffer is free'd in read_complete event above) could result in accessing the
MIOBuffer after it's freed (and is on the free list).
{code}
/home/y/bin/traffic_server(new_IOBufferData_internal(char const*, long,
AllocType)+0x51)[0x4f5bef]
/home/y/bin/traffic_server(IOBufferBlock::alloc(long)+0x2c)[0x4f5eec]
/home/y/bin/traffic_server(MIOBuffer::append_block(long)+0x3a)[0x51dee2]
/home/y/bin/traffic_server(MIOBuffer::add_block()+0x22)[0x51df1a]
/home/y/bin/traffic_server(MIOBuffer::check_add_block()+0x4b)[0x6cbd0b]
/home/y/bin/traffic_server(MIOBuffer::write_avail()+0x18)[0x6cbd86]
/home/y/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*,
EThread*)+0x397)[0x736175]
/home/y/bin/traffic_server(write_to_net(NetHandler*, UnixNetVConnection*,
EThread*)+0x80)[0x735dd7]
/home/y/bin/traffic_server(NetHandler::mainNetEvent(int,
Event*)+0x654)[0x72f5e2]
/home/y/bin/traffic_server(Continuation::handleEvent(int,
void*)+0x6c)[0x4f5ad8]
/home/y/bin/traffic_server(EThread::process_event(Event*, int)+0xc8)[0x756046]
/home/y/bin/traffic_server(EThread::execute()+0x3dc)[0x756550]
/home/y/bin/traffic_server[0x7555c4]
/lib64/libpthread.so.0(+0x3036c079d1)[0x2aeb5b7e89d1]
/lib64/libc.so.6(clone+0x6d)[0x30364e8b6d]
{code}
was (Author: sudheerv):
Further debugging showed that 100 cont implementation calls do_io_write() with
a reader on MIOBuffer (ua_entry-write_buffer). The point where this buffer
(ua_entry-write_buffer) is freed seems to be
VC_EVENT_READ_COMPLETE/HTTP_TUNNEL_EVENT_PRECOMPLETE for the POST data being
read from the client, instead of a WRITE_COMPLETE event for the 100 cont's
do_io_write() operation. This could result in premature free'ing of the buffer,
while the WRITE is not complete yet.
Note how do_io_write/write_to_net (that still has a reference to the _writer of
the 100 cont's MIOBuffer), internally may end up allocating iobuf via
write_avail(). This piece of code (which could get executed after the 100 cont
buffer is free'd in read_complete event above) could result in accessing the
MIOBuffer after it's freed (and is on the free list).
{code}
/home/y/bin/traffic_server(new_IOBufferData_internal(char const*, long,
AllocType)+0x51)[0x4f5bef]
/home/y/bin/traffic_server(IOBufferBlock::alloc(long)+0x2c)[0x4f5eec]
/home/y/bin/traffic_server(MIOBuffer::append_block(long)+0x3a)[0x51dee2]
/home/y/bin/traffic_server(MIOBuffer::add_block()+0x22)[0x51df1a]
/home/y/bin/traffic_server(MIOBuffer::check_add_block()+0x4b)[0x6cbd0b]
/home/y/bin/traffic_server(MIOBuffer::write_avail()+0x18)[0x6cbd86]
/home/y/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*,
EThread*)+0x397)[0x736175]
/home/y/bin/traffic_server(write_to_net(NetHandler*, UnixNetVConnection*,
EThread*)+0x80)[0x735dd7]
/home/y/bin/traffic_server(NetHandler::mainNetEvent(int,
Event*)+0x654)[0x72f5e2]
/home/y/bin/traffic_server(Continuation::handleEvent(int,
void*)+0x6c)[0x4f5ad8]
/home/y/bin/traffic_server(EThread::process_event(Event*, int)+0xc8)[0x756046]
/home/y/bin/traffic_server(EThread::execute()+0x3dc)[0x756550]
/home/y/bin/traffic_server[0x7555c4]
/lib64/libpthread.so.0(+0x3036c079d1)[0x2aeb5b7e89d1]
/lib64/libc.so.6(clone+0x6d)[0x30364e8b6d]
{code}
Seg fault when 100 CONT handling is enabled
---
Key: TS-3285
URL: https://issues.apache.org/jira/browse/TS-3285
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Sudheer Vinukonda
Assignee: Sudheer Vinukonda
Fix For: 5.3.0
With 100 CONT handling enabled in our ats5 production hosts, we are seeing
the below seg fault.
{code}
(gdb) bt
#0 0x00316e432925 in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00316e434105 in abort () at abort.c:92
#2 0x2b6869944458 in ink_die_die_die (retval=1) at ink_error.cc:43
#3
[jira] [Comment Edited] (TS-3285) Seg fault when 100 CONT handling is enabled
[
https://issues.apache.org/jira/browse/TS-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271543#comment-14271543
]
Sudheer Vinukonda edited comment on TS-3285 at 1/9/15 5:20 PM:
---
Further debugging showed that 100 cont implementation calls do_io_write() with
a reader on MIOBuffer (ua_entry-write_buffer). The point where this buffer
(ua_entry-write_buffer) is freed seems to be
VC_EVENT_READ_COMPLETE/HTTP_TUNNEL_EVENT_PRECOMPLETE for the POST data being
read from the client, instead of a WRITE_COMPLETE event for the 100 cont's
do_io_write() operation. This could result in premature free'ing of the buffer,
while the WRITE is not complete yet.
Note how do_io_write/write_to_net (that still has a reference to the _write of
the 100 cont's MIOBuffer), internally may end up allocating iobuf via
write_avail(). This piece of code (which could get executed after the 100 cont
buffer is free'd in read_complete event above) could result in accessing the
MIOBuffer after it's freed (and is on the free list).
{code}
/home/y/bin/traffic_server(new_IOBufferData_internal(char const*, long,
AllocType)+0x51)[0x4f5bef]
/home/y/bin/traffic_server(IOBufferBlock::alloc(long)+0x2c)[0x4f5eec]
/home/y/bin/traffic_server(MIOBuffer::append_block(long)+0x3a)[0x51dee2]
/home/y/bin/traffic_server(MIOBuffer::add_block()+0x22)[0x51df1a]
/home/y/bin/traffic_server(MIOBuffer::check_add_block()+0x4b)[0x6cbd0b]
/home/y/bin/traffic_server(MIOBuffer::write_avail()+0x18)[0x6cbd86]
/home/y/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*,
EThread*)+0x397)[0x736175]
/home/y/bin/traffic_server(write_to_net(NetHandler*, UnixNetVConnection*,
EThread*)+0x80)[0x735dd7]
/home/y/bin/traffic_server(NetHandler::mainNetEvent(int,
Event*)+0x654)[0x72f5e2]
/home/y/bin/traffic_server(Continuation::handleEvent(int,
void*)+0x6c)[0x4f5ad8]
/home/y/bin/traffic_server(EThread::process_event(Event*, int)+0xc8)[0x756046]
/home/y/bin/traffic_server(EThread::execute()+0x3dc)[0x756550]
/home/y/bin/traffic_server[0x7555c4]
/lib64/libpthread.so.0(+0x3036c079d1)[0x2aeb5b7e89d1]
/lib64/libc.so.6(clone+0x6d)[0x30364e8b6d]
{code}
was (Author: sudheerv):
Further debugging showed that 100 cont implementation calls do_io_write() with
a reader on MIOBuffer (ua_entry-write_buffer). The point where this buffer
(ua_entry-write_buffer) is freed seems to be
VC_EVENT_READ_COMPLETE/HTTP_TUNNEL_EVENT_PRECOMPLETE for the POST data being
read, instead of a WRITE_COMPLETE event of the do_io_write() operation. This
could result in premature free'ing of the buffer, while the WRITE is not
complete yet.
Note how do_io_write/write_to_net (that still has a reference to the _write of
the 100 cont's MIOBuffer), internally may end up allocating iobuf via
write_avail(). This piece of code (which could get executed after the 100 cont
buffer is free'd in read_complete event above) could result in accessing the
MIOBuffer after it's freed (and is on the free list).
{code}
/home/y/bin/traffic_server(new_IOBufferData_internal(char const*, long,
AllocType)+0x51)[0x4f5bef]
/home/y/bin/traffic_server(IOBufferBlock::alloc(long)+0x2c)[0x4f5eec]
/home/y/bin/traffic_server(MIOBuffer::append_block(long)+0x3a)[0x51dee2]
/home/y/bin/traffic_server(MIOBuffer::add_block()+0x22)[0x51df1a]
/home/y/bin/traffic_server(MIOBuffer::check_add_block()+0x4b)[0x6cbd0b]
/home/y/bin/traffic_server(MIOBuffer::write_avail()+0x18)[0x6cbd86]
/home/y/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*,
EThread*)+0x397)[0x736175]
/home/y/bin/traffic_server(write_to_net(NetHandler*, UnixNetVConnection*,
EThread*)+0x80)[0x735dd7]
/home/y/bin/traffic_server(NetHandler::mainNetEvent(int,
Event*)+0x654)[0x72f5e2]
/home/y/bin/traffic_server(Continuation::handleEvent(int,
void*)+0x6c)[0x4f5ad8]
/home/y/bin/traffic_server(EThread::process_event(Event*, int)+0xc8)[0x756046]
/home/y/bin/traffic_server(EThread::execute()+0x3dc)[0x756550]
/home/y/bin/traffic_server[0x7555c4]
/lib64/libpthread.so.0(+0x3036c079d1)[0x2aeb5b7e89d1]
/lib64/libc.so.6(clone+0x6d)[0x30364e8b6d]
{code}
Seg fault when 100 CONT handling is enabled
---
Key: TS-3285
URL: https://issues.apache.org/jira/browse/TS-3285
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Sudheer Vinukonda
Fix For: 5.3.0
When 100 CONT handling is enabled in our ats5 production hosts, we are seeing
the below seg fault.
{code}
(gdb) bt
#0 0x00316e432925 in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00316e434105 in abort () at abort.c:92
#2 0x2b6869944458 in ink_die_die_die (retval=1) at ink_error.cc:43
#3 0x2b6869944525 in ink_fatal_va(int, const char *, typedef
__va_list_tag __va_list_tag *)
[jira] [Comment Edited] (TS-3285) Seg fault when 100 CONT handling is enabled
[
https://issues.apache.org/jira/browse/TS-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271543#comment-14271543
]
Sudheer Vinukonda edited comment on TS-3285 at 1/9/15 5:19 PM:
---
Further debugging showed that 100 cont implementation calls do_io_write() with
a reader on MIOBuffer (ua_entry-write_buffer). The point where this buffer
(ua_entry-write_buffer) is freed seems to be
VC_EVENT_READ_COMPLETE/HTTP_TUNNEL_EVENT_PRECOMPLETE for the POST data being
read, instead of a WRITE_COMPLETE event of the do_io_write() operation. This
could result in premature free'ing of the buffer, while the WRITE is not
complete yet.
Note how do_io_write/write_to_net (that still has a reference to the _write of
the 100 cont's MIOBuffer), internally may end up allocating iobuf via
write_avail(). This piece of code (which could get executed after the 100 cont
buffer is free'd in read_complete event above) could result in accessing the
MIOBuffer after it's freed (and is on the free list).
{code}
/home/y/bin/traffic_server(new_IOBufferData_internal(char const*, long,
AllocType)+0x51)[0x4f5bef]
/home/y/bin/traffic_server(IOBufferBlock::alloc(long)+0x2c)[0x4f5eec]
/home/y/bin/traffic_server(MIOBuffer::append_block(long)+0x3a)[0x51dee2]
/home/y/bin/traffic_server(MIOBuffer::add_block()+0x22)[0x51df1a]
/home/y/bin/traffic_server(MIOBuffer::check_add_block()+0x4b)[0x6cbd0b]
/home/y/bin/traffic_server(MIOBuffer::write_avail()+0x18)[0x6cbd86]
/home/y/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*,
EThread*)+0x397)[0x736175]
/home/y/bin/traffic_server(write_to_net(NetHandler*, UnixNetVConnection*,
EThread*)+0x80)[0x735dd7]
/home/y/bin/traffic_server(NetHandler::mainNetEvent(int,
Event*)+0x654)[0x72f5e2]
/home/y/bin/traffic_server(Continuation::handleEvent(int,
void*)+0x6c)[0x4f5ad8]
/home/y/bin/traffic_server(EThread::process_event(Event*, int)+0xc8)[0x756046]
/home/y/bin/traffic_server(EThread::execute()+0x3dc)[0x756550]
/home/y/bin/traffic_server[0x7555c4]
/lib64/libpthread.so.0(+0x3036c079d1)[0x2aeb5b7e89d1]
/lib64/libc.so.6(clone+0x6d)[0x30364e8b6d]
{code}
was (Author: sudheerv):
Further debugging showed that 100 cont implementation calls do_io_write() with
a reader on MIOBuffer (ua_entry-write_buffer). The point where this buffer
(ua_entry-write_buffer) is freed seem sto be
VC_EVENT_READ_COMPLETE/HTTP_TUNNEL_EVENT_PRECOMPLETE for the POST data being
read, instead of a WRITE_COMPLETE event of the do_io_write() operation. This
could result in premature free'ing of the buffer, while the WRITE is not
complete yet.
Note how do_io_write/write_to_net (that still has a reference to the _write of
the 100 cont's MIOBuffer), internally may end up allocating iobuf via
write_avail(). This piece of code (which could get executed after the 100 cont
buffer is free'd in read_complete event above) could result in accessing the
MIOBuffer after it's freed (and is on the free list).
{code}
/home/y/bin/traffic_server(new_IOBufferData_internal(char const*, long,
AllocType)+0x51)[0x4f5bef]
/home/y/bin/traffic_server(IOBufferBlock::alloc(long)+0x2c)[0x4f5eec]
/home/y/bin/traffic_server(MIOBuffer::append_block(long)+0x3a)[0x51dee2]
/home/y/bin/traffic_server(MIOBuffer::add_block()+0x22)[0x51df1a]
/home/y/bin/traffic_server(MIOBuffer::check_add_block()+0x4b)[0x6cbd0b]
/home/y/bin/traffic_server(MIOBuffer::write_avail()+0x18)[0x6cbd86]
/home/y/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*,
EThread*)+0x397)[0x736175]
/home/y/bin/traffic_server(write_to_net(NetHandler*, UnixNetVConnection*,
EThread*)+0x80)[0x735dd7]
/home/y/bin/traffic_server(NetHandler::mainNetEvent(int,
Event*)+0x654)[0x72f5e2]
/home/y/bin/traffic_server(Continuation::handleEvent(int,
void*)+0x6c)[0x4f5ad8]
/home/y/bin/traffic_server(EThread::process_event(Event*, int)+0xc8)[0x756046]
/home/y/bin/traffic_server(EThread::execute()+0x3dc)[0x756550]
/home/y/bin/traffic_server[0x7555c4]
/lib64/libpthread.so.0(+0x3036c079d1)[0x2aeb5b7e89d1]
/lib64/libc.so.6(clone+0x6d)[0x30364e8b6d]
{code}
Seg fault when 100 CONT handling is enabled
---
Key: TS-3285
URL: https://issues.apache.org/jira/browse/TS-3285
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Sudheer Vinukonda
Fix For: 5.3.0
When 100 CONT handling is enabled in our ats5 production hosts, we are seeing
the below seg fault.
{code}
(gdb) bt
#0 0x00316e432925 in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00316e434105 in abort () at abort.c:92
#2 0x2b6869944458 in ink_die_die_die (retval=1) at ink_error.cc:43
#3 0x2b6869944525 in ink_fatal_va(int, const char *, typedef
__va_list_tag __va_list_tag *) (return_code=1,
[jira] [Comment Edited] (TS-3285) Seg fault when 100 CONT handling is enabled
[
https://issues.apache.org/jira/browse/TS-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271543#comment-14271543
]
Sudheer Vinukonda edited comment on TS-3285 at 1/9/15 5:21 PM:
---
Further debugging showed that 100 cont implementation calls do_io_write() with
a reader on MIOBuffer (ua_entry-write_buffer). The point where this buffer
(ua_entry-write_buffer) is freed seems to be
VC_EVENT_READ_COMPLETE/HTTP_TUNNEL_EVENT_PRECOMPLETE for the POST data being
read from the client, instead of a WRITE_COMPLETE event for the 100 cont's
do_io_write() operation. This could result in premature free'ing of the buffer,
while the WRITE is not complete yet.
Note how do_io_write/write_to_net (that still has a reference to the _writer of
the 100 cont's MIOBuffer), internally may end up allocating iobuf via
write_avail(). This piece of code (which could get executed after the 100 cont
buffer is free'd in read_complete event above) could result in accessing the
MIOBuffer after it's freed (and is on the free list).
{code}
/home/y/bin/traffic_server(new_IOBufferData_internal(char const*, long,
AllocType)+0x51)[0x4f5bef]
/home/y/bin/traffic_server(IOBufferBlock::alloc(long)+0x2c)[0x4f5eec]
/home/y/bin/traffic_server(MIOBuffer::append_block(long)+0x3a)[0x51dee2]
/home/y/bin/traffic_server(MIOBuffer::add_block()+0x22)[0x51df1a]
/home/y/bin/traffic_server(MIOBuffer::check_add_block()+0x4b)[0x6cbd0b]
/home/y/bin/traffic_server(MIOBuffer::write_avail()+0x18)[0x6cbd86]
/home/y/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*,
EThread*)+0x397)[0x736175]
/home/y/bin/traffic_server(write_to_net(NetHandler*, UnixNetVConnection*,
EThread*)+0x80)[0x735dd7]
/home/y/bin/traffic_server(NetHandler::mainNetEvent(int,
Event*)+0x654)[0x72f5e2]
/home/y/bin/traffic_server(Continuation::handleEvent(int,
void*)+0x6c)[0x4f5ad8]
/home/y/bin/traffic_server(EThread::process_event(Event*, int)+0xc8)[0x756046]
/home/y/bin/traffic_server(EThread::execute()+0x3dc)[0x756550]
/home/y/bin/traffic_server[0x7555c4]
/lib64/libpthread.so.0(+0x3036c079d1)[0x2aeb5b7e89d1]
/lib64/libc.so.6(clone+0x6d)[0x30364e8b6d]
{code}
was (Author: sudheerv):
Further debugging showed that 100 cont implementation calls do_io_write() with
a reader on MIOBuffer (ua_entry-write_buffer). The point where this buffer
(ua_entry-write_buffer) is freed seems to be
VC_EVENT_READ_COMPLETE/HTTP_TUNNEL_EVENT_PRECOMPLETE for the POST data being
read from the client, instead of a WRITE_COMPLETE event for the 100 cont's
do_io_write() operation. This could result in premature free'ing of the buffer,
while the WRITE is not complete yet.
Note how do_io_write/write_to_net (that still has a reference to the _write of
the 100 cont's MIOBuffer), internally may end up allocating iobuf via
write_avail(). This piece of code (which could get executed after the 100 cont
buffer is free'd in read_complete event above) could result in accessing the
MIOBuffer after it's freed (and is on the free list).
{code}
/home/y/bin/traffic_server(new_IOBufferData_internal(char const*, long,
AllocType)+0x51)[0x4f5bef]
/home/y/bin/traffic_server(IOBufferBlock::alloc(long)+0x2c)[0x4f5eec]
/home/y/bin/traffic_server(MIOBuffer::append_block(long)+0x3a)[0x51dee2]
/home/y/bin/traffic_server(MIOBuffer::add_block()+0x22)[0x51df1a]
/home/y/bin/traffic_server(MIOBuffer::check_add_block()+0x4b)[0x6cbd0b]
/home/y/bin/traffic_server(MIOBuffer::write_avail()+0x18)[0x6cbd86]
/home/y/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*,
EThread*)+0x397)[0x736175]
/home/y/bin/traffic_server(write_to_net(NetHandler*, UnixNetVConnection*,
EThread*)+0x80)[0x735dd7]
/home/y/bin/traffic_server(NetHandler::mainNetEvent(int,
Event*)+0x654)[0x72f5e2]
/home/y/bin/traffic_server(Continuation::handleEvent(int,
void*)+0x6c)[0x4f5ad8]
/home/y/bin/traffic_server(EThread::process_event(Event*, int)+0xc8)[0x756046]
/home/y/bin/traffic_server(EThread::execute()+0x3dc)[0x756550]
/home/y/bin/traffic_server[0x7555c4]
/lib64/libpthread.so.0(+0x3036c079d1)[0x2aeb5b7e89d1]
/lib64/libc.so.6(clone+0x6d)[0x30364e8b6d]
{code}
Seg fault when 100 CONT handling is enabled
---
Key: TS-3285
URL: https://issues.apache.org/jira/browse/TS-3285
Project: Traffic Server
Issue Type: Bug
Components: Core
Affects Versions: 5.0.1
Reporter: Sudheer Vinukonda
Fix For: 5.3.0
When 100 CONT handling is enabled in our ats5 production hosts, we are seeing
the below seg fault.
{code}
(gdb) bt
#0 0x00316e432925 in raise (sig=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00316e434105 in abort () at abort.c:92
#2 0x2b6869944458 in ink_die_die_die (retval=1) at ink_error.cc:43
#3 0x2b6869944525 in ink_fatal_va(int, const char *, typedef
