Re: Segfault in svnserve on UB 16.04 LTS sometimes
Guten Tag Thorsten Schöning,
am Freitag, 12. April 2019 um 16:25 schrieben Sie:
>> StacktraceSource:
>> #0 object_ref_cleanup (baton=0x7f5f75994f00) at
>> /build/subversion-8E3yhQ/subversion-1.9.3/subversion/libsvn_subr/object_pool.c:148
>>143: */
>>144: static apr_status_t
>>145: object_ref_cleanup(void *baton)
>>146: {
>>147: object_ref_t *object = baton;
>>148: svn_object_pool__t *object_pool = object->object_pool;
Any idea on how to progress, e.g. get the operation and repo on which
this happens? Should I post the post to the dev-list instead?
Mit freundlichen Grüßen,
Thorsten Schöning
--
Thorsten Schöning E-Mail: thorsten.schoen...@am-soft.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/
Telefon...05151- 9468- 55
Fax...05151- 9468- 88
Mobil..0178-8 9468- 04
AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
Re: Segfault in svnserve on UB 16.04 LTS sometimes
Guten Tag Stefan Sperling,
am Freitag, 12. April 2019 um 11:45 schrieben Sie:
> Install a package with debug symbols, enable core dumps, and get
> a backtrace with gdb from a core dump.
Does the following help? I've attched some more verbose chrashlog
but without the actual dump, as that is some MiB in size. I can upload
that somewhere if it's considered necessary.
Looks to me like baton has already been freed before without NULLing
and is now illegally accessed again? Still don't know which repo and
operation triggers that.
> Stacktrace:
> #0 object_ref_cleanup (baton=0x7f5f75994f00) at
> /build/subversion-8E3yhQ/subversion-1.9.3/subversion/libsvn_subr/object_pool.c:148
> object = 0x7f5f75994f00
> object_pool =
> #1 0x7f5f747e4e3e in apr_pool_destroy () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> No symbol table info available.
> #2 0x7f5f747e4e15 in apr_pool_destroy () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> No symbol table info available.
> #3 0x00406e4e in main (argc=, argv=0x7ffddc135568)
> at
> /build/subversion-8E3yhQ/subversion-1.9.3/subversion/svnserve/svnserve.c:1368
> pool = 0x7f5f759d6028
> exit_code = 0
> err =
> StacktraceAddressSignature:
> /usr/bin/svnserve:11:/usr/lib/x86_64-linux-gnu/libapr-1.so.0.5.2+1ae3e:/usr/lib/x86_64-linux-gnu/libapr-1.so.0.5.2+1ae15:/usr/bin/svnserve+6e4e
> StacktraceSource:
> #0 object_ref_cleanup (baton=0x7f5f75994f00) at
> /build/subversion-8E3yhQ/subversion-1.9.3/subversion/libsvn_subr/object_pool.c:148
>143: */
>144: static apr_status_t
>145: object_ref_cleanup(void *baton)
>146: {
>147: object_ref_t *object = baton;
>148: svn_object_pool__t *object_pool = object->object_pool;
>149:
>150: /* If we released the last reference to object, there is one more
>151: unused entry.
>152:
>153: Note that unused_count does not need to be always exact but only
> #1 0x7f5f747e4e3e in apr_pool_destroy () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #2 0x7f5f747e4e15 in apr_pool_destroy () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> #3 0x00406e4e in main (argc=, argv=0x7ffddc135568)
> at
> /build/subversion-8E3yhQ/subversion-1.9.3/subversion/svnserve/svnserve.c:1368
>1363: if (threads)
>1364: apr_thread_pool_destroy(threads);
>1365: #endif
>1366:
>1367: /* this will also close the server's socket */
>1368: svn_pool_destroy(pool);
>1369: return exit_code;
>1370: }
>
> StacktraceTop:
> object_ref_cleanup (baton=0x7f5f75994f00) at
> /build/subversion-8E3yhQ/subversion-1.9.3/subversion/libsvn_subr/object_pool.c:148
> apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> apr_pool_destroy () from /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> main (argc=, argv=0x7ffddc135568) at
> /build/subversion-8E3yhQ/subversion-1.9.3/subversion/svnserve/svnserve.c:1368
> ThreadStacktrace:
> .
> Thread 1 (Thread 0x7f5f759c9780 (LWP 3769)):
> #0 object_ref_cleanup (baton=0x7f5f75994f00) at
> /build/subversion-8E3yhQ/subversion-1.9.3/subversion/libsvn_subr/object_pool.c:148
> object = 0x7f5f75994f00
> object_pool =
> #1 0x7f5f747e4e3e in apr_pool_destroy () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> No symbol table info available.
> #2 0x7f5f747e4e15 in apr_pool_destroy () from
> /usr/lib/x86_64-linux-gnu/libapr-1.so.0
> No symbol table info available.
> #3 0x00406e4e in main (argc=, argv=0x7ffddc135568)
> at
> /build/subversion-8E3yhQ/subversion-1.9.3/subversion/svnserve/svnserve.c:1368
> pool = 0x7f5f759d6028
> exit_code = 0
> err =
Some background for at least my docs:
UB 16.04 seems to generate core dumps for this package by default
already, there still was one from the last crash in /var/crash without
me dealing with "ulimit" or whatever[1]:
> cat /proc/sys/kernel/core_pattern
> |/usr/share/apport/apport %p %s %c %d %P
That crashlog contains a stacktrace and core dump at the end, but as
mentioned, the former is not very useful without symbols. Installing
those is easy:
> sudo apt install subversion-dbg
There's some tool called "apport-retrace" easily putting dependencies,
stacktrace with symbols and code etc. into the crashlog:
> sudo apt install apport-retrace
> apport-retrace -R %f
"-R" is important, because for some reason the crashlog misses the
"Package" directive, which makes the app fail.[2] One can either put
that directive into the file manually or use "-R", which does all
those things.[3] While there's the following warning printed in the
end, the overall crashlog looks pretty good:
> W: Can't drop privileges for downloading as file
> 'subversion_1.9.3-2ubuntu1.1.dsc' couldn't be accessed by user '_apt'. -
> pkgAcquire::Run (13: Permission denied)
[1]: https://linux-audit.com/understand-and-configure-core-dumps-work-on-linux/
[2]:
Re: Segfault in svnserve on UB 16.04 LTS sometimes
On Fri, Apr 12, 2019 at 11:42:54AM +0200, Thorsten Schöning wrote: > Does the error tell anyone of you anything already? No. > What should I do > to get some hints about which repo is affected with which operation? Install a package with debug symbols, enable core dumps, and get a backtrace with gdb from a core dump.
Segfault in svnserve on UB 16.04 LTS sometimes
Hi all, today I recognized a segfault in svnserver which seems to have been documented for UB 16.04 LTS already: > Apr 12 09:58:55 [...] kernel: [214930.125762] svnserve[556]: segfault at > 7f5f75994f00 ip 7f5f74ea1065 sp 7ffddc1353f0 error 4 in > libsvn_subr-1.so.1.0.0[7f5f74e43000+d3000] > Apr 12 10:11:41 [...] kernel: [215695.854475] svnserve[3769]: segfault at > 7f5f75994f00 ip 7f5f74ea1065 sp 7ffddc1353f0 error 4 in > libsvn_subr-1.so.1.0.0[7f5f74e43000+d3000] https://answers.launchpad.net/ubuntu/+question/404322 In all cases the version seems to be the default one distirbuted by UB, 1.9.3, and one additional thing in common seems to be the usage of hooks at least in some repos. The thread starter e.g. sends mails, while in one of my repos I'm distributing commits using svnsync. The problem is that it seems a bit difficult to reproduce what triggers the problem: After recognizing it, I committed in some test repo without any hook and in the repo with the hook and in both cases wasn't able to trigger the segfault. Notice that I dind't restart svnserve or the whole system before the tests, just as is. Looking at the past logs, the problem somtimes happens every few minutes, somtimes a few times a day and sometimes even not at all. The interesting thing in my case is that I have some repo to which things get committed multiple times a day automatically. That's as well the repo with the hook. So there shouldn't be any day without any commit to at least one repo, but I still have days without the problem. If the segfaults happen, it seems to be exactly the same error message until the main process gets restarted. My logs contained only one exception from the rule: 7f5f75994640 > Apr 9 12:50:14 [...] kernel: [1677281.956916] svnserve[30221]: segfault at > 7f0258833640 ip 7f0257d40065 sp 7ffe29fd1de0 error 4 in > libsvn_subr-1.so.1.0.0[7f0257ce2000+d3000] > Apr 9 12:55:09 [...] kernel: [1677577.574064] svnserve[31221]: segfault at > 7f0258833640 ip 7f0257d40065 sp 7ffe29fd1de0 error 4 in > libsvn_subr-1.so.1.0.0[7f0257ce2000+d3000] [...] > Apr 10 09:04:14 [...] kernel: [38850.248311] svnserve[19575]: segfault at > 7fe569bcbc30 ip 7fe5690d7065 sp 7ffd951fdad0 error 4 in > libsvn_subr-1.so.1.0.0[7fe569079000+d3000] > Apr 10 09:06:11 [...] kernel: [38967.469929] svnserve[19596]: segfault at > 7f5f75994640 ip 7f5f74ea1065 sp 7ffddc1353f0 error 4 in > libsvn_subr-1.so.1.0.0[7f5f74e43000+d3000] > Apr 10 09:13:25 [...] kernel: [39401.078386] svnserve[20429]: segfault at > 7fe569bcbc30 ip 7fe5690d7065 sp 7ffd951fdad0 error 4 in > libsvn_subr-1.so.1.0.0[7fe569079000+d3000] > Apr 10 09:15:44 [...] kernel: [39539.710149] svnserve[21583]: segfault at > 7fe569bcbc30 ip 7fe5690d7065 sp 7ffd951fdad0 error 4 in > libsvn_subr-1.so.1.0.0[7fe569079000+d3000] [...] > Apr 11 12:07:13 [...] kernel: [136227.892728] svnserve[10466]: segfault at > 7fe569bcbc30 ip 7fe5690d7065 sp 7ffd951fdad0 error 4 in > libsvn_subr-1.so.1.0.0[7fe569079000+d3000] > Apr 11 12:08:38 [...] kernel: [136313.617840] svnserve[10515]: segfault at > 7fe569bcbc30 ip 7fe5690d7065 sp 7ffd951fdad0 error 4 in > libsvn_subr-1.so.1.0.0[7fe569079000+d3000] > Apr 11 12:08:38 [...] kernel: [136313.619646] svnserve[10512]: segfault at > 7fe569bcbc30 ip 7fe5690d7065 sp 7ffd951fdad0 error 4 in > libsvn_subr-1.so.1.0.0[7fe569079000+d3000] > Apr 11 12:09:41 [...] kernel: [136376.041439] svnserve[10591]: segfault at > 7fe569bcbc30 ip 7fe5690d7065 sp 7ffd951fdad0 error 4 in > libsvn_subr-1.so.1.0.0[7fe569079000+d3000] > Apr 11 12:10:10 [...] kernel: [136404.999582] svnserve[10629]: segfault at > 7fe569bcbc30 ip 7fe5690d7065 sp 7ffd951fdad0 error 4 in > libsvn_subr-1.so.1.0.0[7fe569079000+d3000] > Apr 11 12:11:25 [...] kernel: [136480.403404] svnserve[11437]: segfault at > 7fe569bcbc30 ip 7fe5690d7065 sp 7ffd951fdad0 error 4 in > libsvn_subr-1.so.1.0.0[7fe569079000+d3000] > Apr 11 12:11:25 [...] kernel: [136480.414366] svnserve[11438]: segfault at > 7fe569bcbc30 ip 7fe5690d7065 sp 7ffd951fdad0 error 4 in > libsvn_subr-1.so.1.0.0[7fe569079000+d3000] > Apr 11 12:12:05 [...] kernel: [136520.407709] svnserve[11470]: segfault at > 7fe569bcbc30 ip 7fe5690d7065 sp 7ffd951fdad0 error 4 in > libsvn_subr-1.so.1.0.0[7fe569079000+d3000] [...] > Apr 12 09:58:55 [...] kernel: [214930.125762] svnserve[556]: segfault at > 7f5f75994f00 ip 7f5f74ea1065 sp 7ffddc1353f0 error 4 in > libsvn_subr-1.so.1.0.0[7f5f74e43000+d3000] > Apr 12 10:11:41 [...] kernel: [215695.854475] svnserve[3769]: segfault at > 7f5f75994f00 ip 7f5f74ea1065 sp 7ffddc1353f0 error 4 in > libsvn_subr-1.so.1.0.0[7f5f74e43000+d3000] Does the error tell anyone of you anything already? What should I do to get some hints about which repo is affected with which operation? The problem might not be related to commits at all
