Hi Dago and Folks,
On 08/10/2016 07:57 AM, Dagobert Michelsen wrote:
Hi Jake,
(ccing maintainers, hope thats ok)
Am 09.08.2016 um 23:02 schrieb jgoerzen <[email protected]>:
I need some help debugging a segmentation fault in the latest release of
tor. The core file for this program is on the buildfarm here:
/home/jgoerzen/opencsw/tor/trunk/core
If you have time would you have a look at it? I tried using mdb to debug
it but I don't know enough c or assembler to figure out what is going wrong.
Here is what I found running the program with mdb:
jgoerzen@unstable10x [global]:/home/jgoerzen/opencsw/tor/trunk > mdb
work/solaris10-i386/build-isa-pentium_pro/tor-0.2.8.6/src/or/tor
::run
Aug 09 22:00:08.507 [notice] Tor v0.2.8.6 (git-b54676641111fea6) running on
SunOS with Libevent 2.0.21-stable, OpenSSL 1.0.1t and Zlib 1.2.8.
Aug 09 22:00:08.508 [notice] Tor can't help you if you use it wrong! Learn how
to be safe at https://www.torproject.org/download/download#warning
Aug 09 22:00:08.509 [notice] Configuration file "/etc/opt/csw/tor/torrc" not
present, using reasonable defaults.
Aug 09 22:00:08.512 [notice] Opening Socks listener on 127.0.0.1:9050
mdb: stop on SIGSEGV
mdb: target stopped at:
libc.so.1`strlen+0xc: movl (%eax),%edx
mdb: You've got symbols!
Loading modules: [ ld.so.1 libc.so.1 ]
::stack
libc.so.1`strlen+0xc(822dc20, 8047588, 8047450, 0)
libc.so.1`vsnprintf+0x73(804749c, 80, 822dc0a, 8047588, 0, fee95b38)
tor_vasprintf+0x63(82d5138, 822dc0a, 8047588, fee41afc, 8282f08, 82d43c0)
tor_asprintf+0x40(82d5138, 822dc0a, 0, 2346, 0, 0)
dir_server_new+0x288(0, 2346, 2329, 0, 80476e8, 0)
fallback_dir_server_new+0x66(80476bc, 2346, 2329, 0, 80476e8, 0)
parse_dir_fallback_line+0x4c3(82d25f8, 1, 0, 8229a88, 82d25f8, 8282f08)
set_options+0x2475(82d25f8, 8047894, 0, 0, 0, 8047894)
options_init_from_string+0x392(82bbc68, 82bbc78, 0, 0, 8047894, 0)
options_init_from_torrc+0x20f(1, 8047bc8, 820cca8, 8237cc0, 82bc030, 8227dc2)
tor_init+0x380(1, 8047bc8, 8060c30, 8047b80, 0, 0)
tor_main+0x6b(1, 8047bc8, fefc2ab4, feefdeb5, fee73721, 29)
main+0x36(1, 8047bc8, 8047bd0)
_start+0x80(1, 8047cc8, 0, 8047ccc, 8047cd6, 8047ce9)
It looks like it crashes in strlen. So lets see what string it operates on:
0x822dc20/S
0x822dc20: :%d
0x822dc20/4B
0x822dc20: 3a 25 64 0
This looks good. Now what does strlen do?
strlen::dis
libc.so.1`strlen: movl 0x4(%esp),%eax
# Copy arg to register
libc.so.1`strlen+4: testl $0x3,%eax
# adress % 4 == 0 ?
libc.so.1`strlen+9: jne +0x28 <libc.so.1`strlen+0x31>
# No, jump
libc.so.1`strlen+0xb: nop
libc.so.1`strlen+0xc: movl (%eax),%edx
libc.so.1`strlen+0xe: movl $0x7f7f7f7f,%ecx
libc.so.1`strlen+0x13: andl %edx,%ecx
libc.so.1`strlen+0x15: addl $0x4,%eax
libc.so.1`strlen+0x18: addl $0x7f7f7f7f,%ecx
libc.so.1`strlen+0x1e: orl %edx,%ecx
libc.so.1`strlen+0x20: andl $0x80808080,%ecx
libc.so.1`strlen+0x26: cmpl $0x80808080,%ecx
libc.so.1`strlen+0x2c: je -0x20 <libc.so.1`strlen+0xc>
libc.so.1`strlen+0x2e: subl $0x4,%eax
libc.so.1`strlen+0x31: cmpb $0x0,(%eax)
# is the byte zero?
libc.so.1`strlen+0x34: je +0xc <libc.so.1`strlen+0x40>
libc.so.1`strlen+0x36: incl %eax
libc.so.1`strlen+0x37: testl $0x3,%eax
libc.so.1`strlen+0x3c: jne -0xb <libc.so.1`strlen+0x31>
libc.so.1`strlen+0x3e: jmp -0x32 <libc.so.1`strlen+0xc>
libc.so.1`strlen+0x40: subl 0x4(%esp),%eax
libc.so.1`strlen+0x44: ret
This looks like a modern implementation than the usual
size_t strlen(const char *s) {
const char *s0 = s + 1;
while (*s++ != '\0‘)
;
return (s - s0);
}
and pretty similar to
http://cr.illumos.org/~webrev/tsoome/loader_import/usr/src/boot/lib/libc/string/strlen.c.html
So we have SIGSEGV at strlen+0xc, that is an indirect load from %eax, and eax is
::regs
%cs = 0x003b %eax = 0x00000000
…
Trying to read from address 0. This now is strange as 0x822dc20 was passed in
to strlen.
When using lib0@0 there is a related error in tor_asprintf shown, maybe this
leads you
closed to the bug:
dam@unstable10x [global]:/home/dam > LD_PRELOAD_32=/usr/lib/[email protected]
/home/jgoerzen/opencsw/tor/trunk/work/solaris10-i386/build-isa-pentium_pro/tor-0.2.8.6/src/or/tor
Aug 10 16:56:43.295 [notice] Tor v0.2.8.6 (git-b54676641111fea6) running on
SunOS with Libevent 2.0.21-stable, OpenSSL 1.0.1t and Zlib 1.2.8.
Aug 10 16:56:43.295 [notice] Tor can't help you if you use it wrong! Learn how
to be safe at https://www.torproject.org/download/download#warning
Aug 10 16:56:43.339 [notice] Configuration file "/etc/opt/csw/tor/torrc" not
present, using reasonable defaults.
Aug 10 16:56:43.347 [notice] Opening Socks listener on 127.0.0.1:9050
Aug 10 16:56:43.000 [err] tor_asprintf(): Bug: Internal error in asprintf (on
Tor 0.2.8.6 b54676641111fea6)
Aug 10 16:56:43.000 [err] tor_assertion_failed_(): Bug:
src/common/compat.c:529: tor_asprintf: Assertion 0 failed; aborting. (on Tor
0.2.8.6 b54676641111fea6)
Aug 10 16:56:43.000 [err] Bug: Assertion 0 failed in tor_asprintf at
src/common/compat.c:529. (Stack trace not available) (on Tor 0.2.8.6
b54676641111fea6)
At the moment I don’t have a smart idea and would need some more debugging, but
for today
I am out of time, sorry.
Best regards
— Dago
--
"You don't become great by trying to be great, you become great by wanting to
do something,
and then doing it so hard that you become great in the process." - xkcd #896
I notice that the tor_assert(0) is being called because r is being
returned negative. I think it could be in the call to vasprintf()
returning an overflowing buffer looping and becoming a negative number?
Here is a section of the relevant code in src/common/compat.c
/**
* Portable asprintf implementation. Does a printf() into a newly malloc'd
* string. Sets *<b>strp</b> to this string, and returns its length (not
* including the terminating NUL character).
*
* You can treat this function as if its implementation were something like
<pre>
char buf[_INFINITY_];
tor_snprintf(buf, sizeof(buf), fmt, args);
*strp = tor_strdup(buf);
return strlen(*strp):
</pre>
* Where _INFINITY_ is an imaginary constant so big that any string can fit
* into it.
*/
int
tor_asprintf(char **strp, const char *fmt, ...)
{
int r;
va_list args;
va_start(args, fmt);
r = tor_vasprintf(strp, fmt, args);
va_end(args);
if (!*strp || r < 0) {
log_err(LD_BUG, "Internal error in asprintf");
tor_assert(0);
}
return r;
}
/**
* Portable vasprintf implementation. Does a printf() into a newly
malloc'd
* string. Differs from regular vasprintf in the same ways that
* tor_asprintf() differs from regular asprintf.
*/
int
tor_vasprintf(char **strp, const char *fmt, va_list args)
{
/* use a temporary variable in case *strp is in args. */
char *strp_tmp=NULL;
#ifdef HAVE_VASPRINTF
/* If the platform gives us one, use it. */
int r = vasprintf(&strp_tmp, fmt, args);
if (r < 0)
*strp = NULL;
else
*strp = strp_tmp;
return r;
#elif defined(HAVE__VSCPRINTF)
/* On Windows, _vsnprintf won't tell us the length of the string if it
* overflows, so we need to use _vcsprintf to tell how much to
allocate */
int len, r;
len = _vscprintf(fmt, args);
if (len < 0) {
*strp = NULL;
return -1;
}
strp_tmp = tor_malloc(len + 1);
r = _vsnprintf(strp_tmp, len+1, fmt, args);
if (r != len) {
tor_free(strp_tmp);
*strp = NULL;
return -1;
}
*strp = strp_tmp;
return len;
#else
/* Everywhere else, we have a decent vsnprintf that tells us how many
* characters we need. We give it a try on a short buffer first, since
* it might be nice to avoid the second vsnprintf call.
*/
char buf[128];
int len, r;
va_list tmp_args;
va_copy(tmp_args, args);
/* vsnprintf() was properly checked but tor_vsnprintf() available so
* why not use it? */
len = tor_vsnprintf(buf, sizeof(buf), fmt, tmp_args);
va_end(tmp_args);
if (len < (int)sizeof(buf)) {
*strp = tor_strdup(buf);
return len;
}
strp_tmp = tor_malloc(len+1);
/* use of tor_vsnprintf() will ensure string is null terminated */
r = tor_vsnprintf(strp_tmp, len+1, fmt, args);
if (r != len) {
tor_free(strp_tmp);
*strp = NULL;
return -1;
}
*strp = strp_tmp;
return len;
#endif
}
Thanks for your help,
Jake
