Re: segfault with stripped lib, works fine when non-stripped

[Jeremie Le Hen]

Hey,

On Tue, Jan 5, 2016 at 7:38 AM, Stuart Henderson  wrote:
> On 2016-01-04, Jeremie Le Hen  wrote:
>> Hi,
>>
>> Yeah... when you read that subject you probably had this weird gaze ô_Ò
>> like I did when I came to that conclusion.
>>
>> I've been experiencing segfaults in milter-greylist on one of my MX
>> running OpenBSD for a while.  I contacted Stuart (cc'ed) about 6 months
>> ago about this, but gave up because I couldn't manage to compile
>> everything with the debugging symbols.  This time after much struggle to
>> compile the ports chain with them, I finally managed to run
>> milter-greylist in gdb(1) with the hope to witness the live crash and
>> get a detailed stacktrace...
>>
>> Except that even after tinkling Postfix, it never happened.  This simply
>> worked fine.  So after some more tinkering I came to the following
>> conclusion: if I run strip(1) on /usr/local/lib/libbind/libbind.so.5.0
>> to remove the debugging symbols, then it will crash with the stacktrace
>> below.
>
> libbind in the package isn't stripped either, it's just that it isn't built
> with debug symbols. So I'm not sure what's going on there.
>
> Note for anyone else looking: this is all rather dirty because there are
> conflicts between symbols in libc and libbind. It works well enough for
net/mtr
> and for the test program spf_example in libspf2's distribution, but I
suspect
> using it as a milter in the address space of an MTA that's using the libc
> resolver is pushing our luck too far.

Alright, I finally got some time (vacations) to fix this. I got rid of
the problem by removing the actually useless dependency over libbind.
The binary has been running for one hour without crashing which was
impossible previously.

See the following patches for mail/libspf2 and mail/milter-greylist (I
cc'ed Jakob, the maintainer):

https://people.freebsd.org/~jlh/openbsd_mail_libspf2_no_libbind.diff
https://people.freebsd.org/~jlh/openbsd_mail_milter-greylist_no_libbind.diff


>
>
>> #0  0x1cc53e386d40 in memcpy (dst0=0x1cc5c48b7000, src0=Variable "src0"
is not available.
>> ) at /usr/src/lib/libc/string/memcpy.c:94
>> #1  0x1cc4f4d496d8 in __res_vinit () from
/usr/local/lib/libbind/libbind.so.5.0
>> #2  0x1cc4f4d48bda in __res_ninit () from
/usr/local/lib/libbind/libbind.so.5.0
>> #3  0x1cc50b181905 in SPF_dns_resolv_lookup
(spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org",
rr_type=ns_t_spf, should_cache=1) at spf_dns_resolv.c:261
>> #4  0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48ab780,
domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at
spf_dns.c:141
>> #5  0x1cc50b180b16 in SPF_dns_cache_lookup
(spf_dns_server=0x1cc5c48abc80, domain=0x1cc55122c1d0 "mydomain.org",
rr_type=ns_t_spf, should_cache=1) at spf_dns_cache.c:408
>> #6  0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48abc80,
domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at
spf_dns.c:141
>> #7  0x1cc50b18e4e3 in SPF_server_get_record (spf_server=0x1cc5eb4154c0,
spf_request=0x1cc5c48aeb00, spf_response=0x1cc5eb41b400,
spf_recordp=0x1cc54f7c8700) at spf_server.c:351
>> #8  0x1cc50b18c959 in SPF_request_query_mailfrom
(spf_request=0x1cc5c48aeb00, spf_responsep=0x1cc54f7c87a0) at
spf_request.c:291
>> #9  0x1cc2ee1207ca in spf_check_internal (ad=0x1cc4f4c65948,
as=AS_RCPT, ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:388
>> #10 0x1cc2ee120c17 in spf_check (ad=0x1cc4f4c65948, as=AS_RCPT,
ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:524
>> #11 0x1cc2ee123a0d in acl_filter (stage=AS_RCPT, ctx=0x1cc5c48b2000,
priv=0x1cc5c48af000) at acl.c:1902
>> #12 0x1cc2ee1069ae in real_envrcpt (ctx=0x1cc5c48b2000,
envrcpt=0x1cc5eb41c280) at milter-greylist.c:601
>> #13 0x1cc2ee105de0 in mlfi_envrcpt (ctx=0x1cc5c48b2000,
envrcpt=0x1cc5eb41c280) at milter-greylist.c:213
>> #14 0x1cc52bfaa46e in st_rcpt () from /usr/local/lib/libmilter.so.4.0
>> #15 0x1cc52bfab557 in mi_engine () from
/usr/local/lib/libmilter.so.4.0
>> #16 0x1cc52bfaca10 in mi_handle_session () from
/usr/local/lib/libmilter.so.4.0
>> #17 0x1cc52bfab7d9 in mi_thread_handle_wrapper () from
/usr/local/lib/libmilter.so.4.0
>> #18 0x1cc5a247d90e in _rthread_start (v=Variable "v" is not available.
>> ) at /usr/src/lib/librthread/rthread.c:145
>> #19 0x1cc53e33649b in __tfork_thread () at
/usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:75
>> #20 0x in ?? ()
>


--
Jeremie Le Hen
j...@freebsd.org

Re: segfault with stripped lib, works fine when non-stripped

[Boudewijn Dijkstra]

Op Mon, 04 Jan 2016 22:54:24 +0100 schreef Jeremie Le Hen :

Yeah... when you read that subject you probably had this weird gaze ô_Ò
like I did when I came to that conclusion.

I've been experiencing segfaults in milter-greylist [...]
 So after some more tinkering I came to the following
conclusion: if I run strip(1) on /usr/local/lib/libbind/libbind.so.5.0
to remove the debugging symbols, then it will crash with the stacktrace
below.

Has anyone of you seen such a behavior in the past?

#0  0x1cc53e386d40 in memcpy (dst0=0x1cc5c48b7000, src0=Variable "src0"

is not available.

) at /usr/src/lib/libc/string/memcpy.c:94
#1  0x1cc4f4d496d8 in __res_vinit () from

/usr/local/lib/libbind/libbind.so.5.0

#2  0x1cc4f4d48bda in __res_ninit () from

/usr/local/lib/libbind/libbind.so.5.0

#3  0x1cc50b181905 in SPF_dns_resolv_lookup

(spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org",
rr_type=ns_t_spf, should_cache=1) at spf_dns_resolv.c:261

[...]


I had the same stack trace with a simple test application
(http://www.libspf2.org/docs/html/ ). Except that my memcpy showed src=0,
len=16.  OpenBSD 5.8 i386 with stock libs (from packages). The prebuilt tools
(in /usr/local/libexec/libspf2/) ran fine. My Makefile had:
LDADD+= -lspf2

Segfaults disappeared when I changed that to:
LDADD+= -lbind -lspf2

So yeah, looks like conflicts between symbols.


--
(Remove the obvious prefix to reply privately.)
Gemaakt met Opera's e-mailprogramma: http://www.opera.com/mail/

Re: segfault with stripped lib, works fine when non-stripped

[Joel Rees]

On Tue, Jan 5, 2016 at 6:54 AM, Jeremie Le Hen  wrote:
> Hi,
>
> Yeah... when you read that subject you probably had this weird gaze ô_Ò
> like I did when I came to that conclusion.

Theo will probably get after me for responding when I don't know what
I'm talking about, and I don't have specific experience with this
package, but it's not unusual, in general.

Stripping symbols can reveal lots of things where prorammers have cut
corners or just made mistakes -- pointer issues are especially
prominent but not the only problems. However, in this case, I'm
guessing it would probably not be pointer issues so much as something
like allocation class issues that are hidden when the symbols are left
in, but exposed when the symbols are stripped.

That said, you may not want to think too hard about that just yet.

> I've been experiencing segfaults in milter-greylist on one of my MX
> running OpenBSD for a while.  I contacted Stuart (cc'ed) about 6 months
> ago about this, but gave up because I couldn't manage to compile
> everything with the debugging symbols.  This time after much struggle to
> compile the ports chain with them, I finally managed to run
> milter-greylist in gdb(1) with the hope to witness the live crash and
> get a detailed stacktrace...
>
> Except that even after tinkling Postfix, it never happened.  This simply
> worked fine.  So after some more tinkering I came to the following
> conclusion: if I run strip(1) on /usr/local/lib/libbind/libbind.so.5.0
> to remove the debugging symbols, then it will crash with the stacktrace
> below.
>
> Has anyone of you seen such a behavior in the past?
>
> #0  0x1cc53e386d40 in memcpy (dst0=0x1cc5c48b7000, src0=Variable "src0"
is not available.

Do you have any idea why "src0" isn't available here?

That might be a good place to start.

> ) at /usr/src/lib/libc/string/memcpy.c:94
> #1  0x1cc4f4d496d8 in __res_vinit () from
/usr/local/lib/libbind/libbind.so.5.0
> #2  0x1cc4f4d48bda in __res_ninit () from
/usr/local/lib/libbind/libbind.so.5.0
> #3  0x1cc50b181905 in SPF_dns_resolv_lookup
(spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org",
rr_type=ns_t_spf, should_cache=1) at spf_dns_resolv.c:261
> #4  0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48ab780,
domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at
spf_dns.c:141
> #5  0x1cc50b180b16 in SPF_dns_cache_lookup
(spf_dns_server=0x1cc5c48abc80, domain=0x1cc55122c1d0 "mydomain.org",
rr_type=ns_t_spf, should_cache=1) at spf_dns_cache.c:408
> #6  0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48abc80,
domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at
spf_dns.c:141
> #7  0x1cc50b18e4e3 in SPF_server_get_record (spf_server=0x1cc5eb4154c0,
spf_request=0x1cc5c48aeb00, spf_response=0x1cc5eb41b400,
spf_recordp=0x1cc54f7c8700) at spf_server.c:351
> #8  0x1cc50b18c959 in SPF_request_query_mailfrom
(spf_request=0x1cc5c48aeb00, spf_responsep=0x1cc54f7c87a0) at
spf_request.c:291
> #9  0x1cc2ee1207ca in spf_check_internal (ad=0x1cc4f4c65948, as=AS_RCPT,
ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:388
> #10 0x1cc2ee120c17 in spf_check (ad=0x1cc4f4c65948, as=AS_RCPT,
ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:524
> #11 0x1cc2ee123a0d in acl_filter (stage=AS_RCPT, ctx=0x1cc5c48b2000,
priv=0x1cc5c48af000) at acl.c:1902
> #12 0x1cc2ee1069ae in real_envrcpt (ctx=0x1cc5c48b2000,
envrcpt=0x1cc5eb41c280) at milter-greylist.c:601
> #13 0x1cc2ee105de0 in mlfi_envrcpt (ctx=0x1cc5c48b2000,
envrcpt=0x1cc5eb41c280) at milter-greylist.c:213
> #14 0x1cc52bfaa46e in st_rcpt () from /usr/local/lib/libmilter.so.4.0
> #15 0x1cc52bfab557 in mi_engine () from /usr/local/lib/libmilter.so.4.0
> #16 0x1cc52bfaca10 in mi_handle_session () from
/usr/local/lib/libmilter.so.4.0
> #17 0x1cc52bfab7d9 in mi_thread_handle_wrapper () from
/usr/local/lib/libmilter.so.4.0
> #18 0x1cc5a247d90e in _rthread_start (v=Variable "v" is not available.
> ) at /usr/src/lib/librthread/rthread.c:145
> #19 0x1cc53e33649b in __tfork_thread () at
/usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:75
> #20 0x in ?? ()
>
>
> --
> Jeremie Le Hen
> j...@freebsd.org
>



--
Joel Rees

Be careful when you look at conspiracy.
Arm yourself with knowledge of yourself, as well:
http://reiisi.blogspot.jp/2011/10/conspiracy-theories.html

segfault with stripped lib, works fine when non-stripped

[Jeremie Le Hen]

Hi,

Yeah... when you read that subject you probably had this weird gaze ô_Ò 
like I did when I came to that conclusion.

I've been experiencing segfaults in milter-greylist on one of my MX
running OpenBSD for a while.  I contacted Stuart (cc'ed) about 6 months
ago about this, but gave up because I couldn't manage to compile
everything with the debugging symbols.  This time after much struggle to
compile the ports chain with them, I finally managed to run
milter-greylist in gdb(1) with the hope to witness the live crash and
get a detailed stacktrace...

Except that even after tinkling Postfix, it never happened.  This simply 
worked fine.  So after some more tinkering I came to the following
conclusion: if I run strip(1) on /usr/local/lib/libbind/libbind.so.5.0
to remove the debugging symbols, then it will crash with the stacktrace
below.

Has anyone of you seen such a behavior in the past?

#0  0x1cc53e386d40 in memcpy (dst0=0x1cc5c48b7000, src0=Variable "src0" is 
not available.
) at /usr/src/lib/libc/string/memcpy.c:94
#1  0x1cc4f4d496d8 in __res_vinit () from 
/usr/local/lib/libbind/libbind.so.5.0
#2  0x1cc4f4d48bda in __res_ninit () from 
/usr/local/lib/libbind/libbind.so.5.0
#3  0x1cc50b181905 in SPF_dns_resolv_lookup (spf_dns_server=0x1cc5c48ab780, 
domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at 
spf_dns_resolv.c:261
#4  0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48ab780, 
domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at 
spf_dns.c:141
#5  0x1cc50b180b16 in SPF_dns_cache_lookup (spf_dns_server=0x1cc5c48abc80, 
domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at 
spf_dns_cache.c:408
#6  0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48abc80, 
domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at 
spf_dns.c:141
#7  0x1cc50b18e4e3 in SPF_server_get_record (spf_server=0x1cc5eb4154c0, 
spf_request=0x1cc5c48aeb00, spf_response=0x1cc5eb41b400, 
spf_recordp=0x1cc54f7c8700) at spf_server.c:351
#8  0x1cc50b18c959 in SPF_request_query_mailfrom 
(spf_request=0x1cc5c48aeb00, spf_responsep=0x1cc54f7c87a0) at spf_request.c:291
#9  0x1cc2ee1207ca in spf_check_internal (ad=0x1cc4f4c65948, as=AS_RCPT, 
ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:388
#10 0x1cc2ee120c17 in spf_check (ad=0x1cc4f4c65948, as=AS_RCPT, 
ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:524
#11 0x1cc2ee123a0d in acl_filter (stage=AS_RCPT, ctx=0x1cc5c48b2000, 
priv=0x1cc5c48af000) at acl.c:1902
#12 0x1cc2ee1069ae in real_envrcpt (ctx=0x1cc5c48b2000, 
envrcpt=0x1cc5eb41c280) at milter-greylist.c:601
#13 0x1cc2ee105de0 in mlfi_envrcpt (ctx=0x1cc5c48b2000, 
envrcpt=0x1cc5eb41c280) at milter-greylist.c:213
#14 0x1cc52bfaa46e in st_rcpt () from /usr/local/lib/libmilter.so.4.0
#15 0x1cc52bfab557 in mi_engine () from /usr/local/lib/libmilter.so.4.0
#16 0x1cc52bfaca10 in mi_handle_session () from 
/usr/local/lib/libmilter.so.4.0
#17 0x1cc52bfab7d9 in mi_thread_handle_wrapper () from 
/usr/local/lib/libmilter.so.4.0
#18 0x1cc5a247d90e in _rthread_start (v=Variable "v" is not available.
) at /usr/src/lib/librthread/rthread.c:145
#19 0x1cc53e33649b in __tfork_thread () at 
/usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:75
#20 0x in ?? ()


-- 
Jeremie Le Hen
j...@freebsd.org

Re: segfault with stripped lib, works fine when non-stripped

[Gregor Best]

Hi Jeremie,

On Mon, Jan 04, 2016 at 10:54:24PM +0100, Jeremie Le Hen wrote:
> [...]
> Has anyone of you seen such a behavior in the past?
> [...]

Haven't seen something like that but my next step would be to build it
with CFLAGS="-g -O0" and without stripping for maximum debuggability and
run it in valgrind to see if there are weird memory access patterns.

-- 
Gregor

Re: segfault with stripped lib, works fine when non-stripped

[Stuart Henderson]

On 2016-01-04, Jeremie Le Hen  wrote:
> Hi,
>
> Yeah... when you read that subject you probably had this weird gaze ô_Ò 
> like I did when I came to that conclusion.
>
> I've been experiencing segfaults in milter-greylist on one of my MX
> running OpenBSD for a while.  I contacted Stuart (cc'ed) about 6 months
> ago about this, but gave up because I couldn't manage to compile
> everything with the debugging symbols.  This time after much struggle to
> compile the ports chain with them, I finally managed to run
> milter-greylist in gdb(1) with the hope to witness the live crash and
> get a detailed stacktrace...
>
> Except that even after tinkling Postfix, it never happened.  This simply 
> worked fine.  So after some more tinkering I came to the following
> conclusion: if I run strip(1) on /usr/local/lib/libbind/libbind.so.5.0
> to remove the debugging symbols, then it will crash with the stacktrace
> below.

libbind in the package isn't stripped either, it's just that it isn't built
with debug symbols. So I'm not sure what's going on there.

Note for anyone else looking: this is all rather dirty because there are
conflicts between symbols in libc and libbind. It works well enough for net/mtr
and for the test program spf_example in libspf2's distribution, but I suspect
using it as a milter in the address space of an MTA that's using the libc
resolver is pushing our luck too far.


> #0  0x1cc53e386d40 in memcpy (dst0=0x1cc5c48b7000, src0=Variable "src0" 
> is not available.
> ) at /usr/src/lib/libc/string/memcpy.c:94
> #1  0x1cc4f4d496d8 in __res_vinit () from 
> /usr/local/lib/libbind/libbind.so.5.0
> #2  0x1cc4f4d48bda in __res_ninit () from 
> /usr/local/lib/libbind/libbind.so.5.0
> #3  0x1cc50b181905 in SPF_dns_resolv_lookup 
> (spf_dns_server=0x1cc5c48ab780, domain=0x1cc55122c1d0 "mydomain.org", 
> rr_type=ns_t_spf, should_cache=1) at spf_dns_resolv.c:261
> #4  0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48ab780, 
> domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at 
> spf_dns.c:141
> #5  0x1cc50b180b16 in SPF_dns_cache_lookup 
> (spf_dns_server=0x1cc5c48abc80, domain=0x1cc55122c1d0 "mydomain.org", 
> rr_type=ns_t_spf, should_cache=1) at spf_dns_cache.c:408
> #6  0x1cc50b180117 in SPF_dns_lookup (spf_dns_server=0x1cc5c48abc80, 
> domain=0x1cc55122c1d0 "mydomain.org", rr_type=ns_t_spf, should_cache=1) at 
> spf_dns.c:141
> #7  0x1cc50b18e4e3 in SPF_server_get_record (spf_server=0x1cc5eb4154c0, 
> spf_request=0x1cc5c48aeb00, spf_response=0x1cc5eb41b400, 
> spf_recordp=0x1cc54f7c8700) at spf_server.c:351
> #8  0x1cc50b18c959 in SPF_request_query_mailfrom 
> (spf_request=0x1cc5c48aeb00, spf_responsep=0x1cc54f7c87a0) at 
> spf_request.c:291
> #9  0x1cc2ee1207ca in spf_check_internal (ad=0x1cc4f4c65948, as=AS_RCPT, 
> ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:388
> #10 0x1cc2ee120c17 in spf_check (ad=0x1cc4f4c65948, as=AS_RCPT, 
> ap=0x1cc54f7c8cd0, priv=0x1cc5c48af000) at spf.c:524
> #11 0x1cc2ee123a0d in acl_filter (stage=AS_RCPT, ctx=0x1cc5c48b2000, 
> priv=0x1cc5c48af000) at acl.c:1902
> #12 0x1cc2ee1069ae in real_envrcpt (ctx=0x1cc5c48b2000, 
> envrcpt=0x1cc5eb41c280) at milter-greylist.c:601
> #13 0x1cc2ee105de0 in mlfi_envrcpt (ctx=0x1cc5c48b2000, 
> envrcpt=0x1cc5eb41c280) at milter-greylist.c:213
> #14 0x1cc52bfaa46e in st_rcpt () from /usr/local/lib/libmilter.so.4.0
> #15 0x1cc52bfab557 in mi_engine () from /usr/local/lib/libmilter.so.4.0
> #16 0x1cc52bfaca10 in mi_handle_session () from 
> /usr/local/lib/libmilter.so.4.0
> #17 0x1cc52bfab7d9 in mi_thread_handle_wrapper () from 
> /usr/local/lib/libmilter.so.4.0
> #18 0x1cc5a247d90e in _rthread_start (v=Variable "v" is not available.
> ) at /usr/src/lib/librthread/rthread.c:145
> #19 0x1cc53e33649b in __tfork_thread () at 
> /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:75
> #20 0x in ?? ()