Re: [EXTERNAL] Re: svn checkout Hangs/Crashes/Succeeds Over HTTP
Den mån 27 maj 2024 kl 14:18 skrev Johan Corveleyn :
> On Sat, May 25, 2024 at 12:12 AM Williams, James P. {Jim}
> (JSC-CD4)[KBR Wyle Services, LLC] via users
> wrote:
> >
> > > Den lör 11 maj 2024 kl 03:00 skrev Williams, James P. {Jim}
> (JSC-CD4)[KBR Wyle Services, LLC] via users :
> >
> > > You previously mentioned Subversion 1.14.1, is that on the server or
> on the client?
> >
> > I'm using 1.14.1 on both the client and server.
> >
> > > Still it would be interesting to compare just to rule out a problem
> within the repository. You can use svnserve directly or tunneled over SSH,
> see the Subversion book:
> >
> > With svnserve 1.14.1, I see no problems. Checkouts complete every
> time. I'm not sure what to conclude about that. It's a different
> protocol, so it doesn't necessarily exonerate the client or the network.
> >
> > > > #0 epoll_wait /usr/lib64/libc.so.6
> >
> > > Waiting for a reply from the server ... ?
> >
> > Yeah, that'd be my guess. When the hang occurs, I've got about 90% of
> the working copy checked out. I expect the client is waiting for more
> bytes to arrive on the socket.
> >
> > > Do you see any activity on the server (CPU / disk) during this time?
> >
> > The server is well-behaved throughout all of my tests. It shows no CPU
> spike or log messages hinting that it's noticed a problem.
>
> That's why my bet is still on "something between client and server"
> (proxy, reverse-proxy, security scanning soft, ...) that messes with
> the network transfer (http or https). That would explain the symptoms
> you're seeing (client hangs waiting for network (and sometimes
> crashes), server has nothing to do and doesn't report anything
> special).
>
I agree with Johan and I understand it might be hard to nail down the
actual issue.
I don't think I have asked this before:
- Can you log in to the server and do an svn checkout https://localhost/[...]
locally and does this succeed? You might have to update the Apache HTTPD
configuration to allow the vhost to reply to "localhost" (or you could add
the server name to the local hosts file pointing to 127.0.0.1 and do a
checkout of https://[server]/[url]). What I'd like to accomplish is a
checkout that doesn't leave the machine. Does this make a difference?
Kind regards,
Daniel
Re: [EXTERNAL] Re: svn checkout Hangs/Crashes/Succeeds Over HTTP
On Sat, May 25, 2024 at 12:12 AM Williams, James P. {Jim}
(JSC-CD4)[KBR Wyle Services, LLC] via users
wrote:
>
> > Den lör 11 maj 2024 kl 03:00 skrev Williams, James P. {Jim} (JSC-CD4)[KBR
> > Wyle Services, LLC] via users :
>
> > You previously mentioned Subversion 1.14.1, is that on the server or on the
> > client?
>
> I'm using 1.14.1 on both the client and server.
>
> > Still it would be interesting to compare just to rule out a problem within
> > the repository. You can use svnserve directly or tunneled over SSH, see the
> > Subversion book:
>
> With svnserve 1.14.1, I see no problems. Checkouts complete every time. I'm
> not sure what to conclude about that. It's a different protocol, so it
> doesn't necessarily exonerate the client or the network.
>
> > > #0 epoll_wait /usr/lib64/libc.so.6
>
> > Waiting for a reply from the server ... ?
>
> Yeah, that'd be my guess. When the hang occurs, I've got about 90% of the
> working copy checked out. I expect the client is waiting for more bytes to
> arrive on the socket.
>
> > Do you see any activity on the server (CPU / disk) during this time?
>
> The server is well-behaved throughout all of my tests. It shows no CPU spike
> or log messages hinting that it's noticed a problem.
That's why my bet is still on "something between client and server"
(proxy, reverse-proxy, security scanning soft, ...) that messes with
the network transfer (http or https). That would explain the symptoms
you're seeing (client hangs waiting for network (and sometimes
crashes), server has nothing to do and doesn't report anything
special).
--
Johan
RE: [EXTERNAL] Re: svn checkout Hangs/Crashes/Succeeds Over HTTP
> Den lör 11 maj 2024 kl 03:00 skrev Williams, James P. {Jim} (JSC-CD4)[KBR
> Wyle Services, LLC] via users :
> You previously mentioned Subversion 1.14.1, is that on the server or on the
> client?
I'm using 1.14.1 on both the client and server.
> Still it would be interesting to compare just to rule out a problem within
> the repository. You can use svnserve directly or tunneled over SSH, see the
> Subversion book:
With svnserve 1.14.1, I see no problems. Checkouts complete every time. I'm
not sure what to conclude about that. It's a different protocol, so it doesn't
necessarily exonerate the client or the network.
> > #0 epoll_wait /usr/lib64/libc.so.6
> Waiting for a reply from the server ... ?
Yeah, that'd be my guess. When the hang occurs, I've got about 90% of the
working copy checked out. I expect the client is waiting for more bytes to
arrive on the socket.
> Do you see any activity on the server (CPU / disk) during this time?
The server is well-behaved throughout all of my tests. It shows no CPU spike
or log messages hinting that it's noticed a problem.
> Memory allocation?
Yeah, both forms of core dumps I've seen have memory/pool allocation at the top
of the stack. Maybe some odd reentrancy case is being tickled that's not often
seen. It points at a likely secondary problem, a bug in the client.
> Parsing the XML message from the server?
> Can you catch/view the actual XML message sent from the server? I'm thinking
> if this is mangled in some strange way that is upsetting the XML parser.
We're not able to install tools like wireshark, if that's what you're
suggesting. I don't see a way to get to that XML other than doctoring SVN
source.
> Again something with memory allocation - same here, can you see what the
> server is actually sending?
Same answer.
> I don't immediately see the call stacks above and the fact that it would fail
> more often if the WC is on an NFS drive. Possibly if the NFS drive is slower
> and this causes some kind of timeout? Can you create a ramdisk and have the
> WC there temporary and see if there is a difference?
I think NFS definitely slows things down, and that change in timing makes the
hangs and crashes more likely. Unfortunately, I don't have the access needed
to create a ramdisk. I'm able to checkout onto a local or NFS-mounted disk
though. I would think the former is equivalent. No?
Thanks for the reply, Daniel.
Jim
From: Daniel Sahlberg
Sent: Saturday, May 11, 2024 1:51 PM
To: Williams, James P. {Jim} (JSC-CD4)[KBR Wyle Services, LLC]
Cc: users@subversion.apache.org
Subject: Re: [EXTERNAL] Re: svn checkout Hangs/Crashes/Succeeds Over HTTP
CAUTION: This email originated from outside of NASA. Please take care when
clicking links or opening attachments. Use the "Report Message" button to
report suspicious messages to the NASA SOC.
Hi,
I've added a few comments/questions below.
Kind regards,
Daniel Sahlberg
Den lör 11 maj 2024 kl 03:00 skrev Williams, James P. {Jim} (JSC-CD4)[KBR Wyle
Services, LLC] via users
mailto:users@subversion.apache.org>>:
> How did you upgrade your server from RHEL 6 to RHEL 8?
Because so much changed from RHEL 6 to 8, including Apache from 2.2.15 to
2.4.37, all the Apache modules, etc., I started from the skeleton configuration
the operating system provides and made mostly the same customizations we had
for RHEL 6, or modernized them where the docs said things changed. Mostly,
that was tweaks to authentication (from LDAP to Kerberos), SSL, and the SVN
endpoints. Browser access to all SVN and ViewVC pages seems to work fine.
You previously mentioned Subversion 1.14.1, is that on the server or on the
client?
[...]
> And do the problems happen if you use svn:// rather than https:// ?
I thought svn:// worked only with svnserve, which we don't run. Are you
suggesting I try to run it as a test, or that I consider abandoning Apache in
favor of it? Yikes; that'd be painful.
I hear you on the HTTP integration. We have about 2000 repos and a few hundred
developers. I've supported that server for at least 15 years, and it hasn't
been too bad...until now.
I have personally only ever used Subversion over http/https (except for testing
purposes) and I haven't had any of the problems described by Nico - I guess
YMMV...
Still it would be interesting to compare just to rule out a problem within the
repository. You can use svnserve directly or tunneled over SSH, see the
Subversion book:
https://svnbook.red-bean.com/en/1.7/svn.serverconfig.svnserve.html#svn.serverconfig.svnserve.sshauth
On Fri, May 10, 2024 at 4:17 PM Williams, James P. {Jim} (JSC-CD4)[KBR
Wyle Services, LLC] via users
mailto:users@subversion.apache.org>> wrote:
>
> I'm upgrading an Apache HTTP server of our SVN repos on RedHat Enterprise
> Linux 8. Using Subversion 1
Re: [EXTERNAL] Re: svn checkout Hangs/Crashes/Succeeds Over HTTP
Den fre 17 maj 2024 kl 00:20 skrev Williams, James P. {Jim} (JSC-CD4)[KBR
Wyle Services, LLC] via users :
> > BTW: as Daniel asked: You previously mentioned Subversion 1.14.1, is
> > that on the server or on the client?
>
> I've been testing with SVN 1.14.1 on both the client and server.
>
Perfect, thanks for confirming. Do you happen to know what version of Serf
is used on the client and what version of OpenSSL do you have on the
client/server?
>
> > Regardless, since the issue manifests on the client-side by hangs and
> > crashes while waiting for / processing data from the server (hangs
> > inside libsvn_ra_serf), I'd suggest also to investigate whether there
> > is something in between your client and your server that might be
> > interfering. A proxy or reverse proxy perhaps? Or some security
> > software on the client that interferes with the network (as some
> > antivirus suites do on Windows)? If so, try to bypass it (or disable /
> > create an exclude rule), as a diagnostic step to see whether this
> > might be the cause.
>
> I don't think we have anything inserted between client and server, but
> will ask those smarter than me if they know of such a thing.
>
> I do have an update. My system administrator was surprised to see the
> server machine configured to support both version 4 and 6 IP addresses. He
> thought the latter were turned off. After that change, checkouts were
> noticeably faster, and hangs and crashes were noticeably less frequent.
> The repo size needed to trigger them appeared to grow some. Though a step
> forward, the configuration still isn't useable. It might also point at
> further network configuration problems as a cause, though the SVN client
> probably shouldn't crash in any case.
>
Are the client and the server both "close" (networkwise) or do they
communicate over a WAN/internet link?
Is it possible to - at least temporarily - add a VHost without SSL
encryption to see if this makes a difference? That would also make it
easier to catch the traffic using Wireshark to see the last packages before
a crash. In particular, I'm interested in the content of the packages that
trigger the xml parsing error (the call to svn_xml_parse). For the cases
when it hangs, does one side say "goodbye" (ie, FIN) or does it seem like
package drops (ie, traffic just stops flowing).
When it hangs, does it ever release? I would assume that the client would
finally time out.
Kind regards,
Daniel
RE: [EXTERNAL] Re: svn checkout Hangs/Crashes/Succeeds Over HTTP
> > > > I've tried with multiple repos of different sizes and ages. The > > > smaller repo I mentioned has about 150 files in trunk, mostly 50 KB > or > > > smaller, and about 500 revisions. A larger repo with the same > problems > > > has about 5000 files in trunk and 10,000 revisions. > > > > > > That *hints* at an httpd tuning issue, but I'm not sure. Check the > httpd > > > logs? > > > > The httpd logs show no signs of a problem. Success and failure cases > look the same in the logs. > > In your initial post your mentioned: > > >>> svn 1.10.2 was failing the same way before we upgraded to 1.14.1 as > a possible fix. > > So the problem isn't new after the upgrade. Agreed. I see no change in behavior, good or bad, between SVN 1.10.2 and 1.14.1 for the many tests I've run. > BTW: as Daniel asked: You previously mentioned Subversion 1.14.1, is > that on the server or on the client? I've been testing with SVN 1.14.1 on both the client and server. > Regardless, since the issue manifests on the client-side by hangs and > crashes while waiting for / processing data from the server (hangs > inside libsvn_ra_serf), I'd suggest also to investigate whether there > is something in between your client and your server that might be > interfering. A proxy or reverse proxy perhaps? Or some security > software on the client that interferes with the network (as some > antivirus suites do on Windows)? If so, try to bypass it (or disable / > create an exclude rule), as a diagnostic step to see whether this > might be the cause. I don't think we have anything inserted between client and server, but will ask those smarter than me if they know of such a thing. I do have an update. My system administrator was surprised to see the server machine configured to support both version 4 and 6 IP addresses. He thought the latter were turned off. After that change, checkouts were noticeably faster, and hangs and crashes were noticeably less frequent. The repo size needed to trigger them appeared to grow some. Though a step forward, the configuration still isn't useable. It might also point at further network configuration problems as a cause, though the SVN client probably shouldn't crash in any case. Thanks for the reply. Jim
Re: [EXTERNAL] Re: svn checkout Hangs/Crashes/Succeeds Over HTTP
On Wed, May 15, 2024 at 12:41 AM Williams, James P. {Jim}
(JSC-CD4)[KBR Wyle Services, LLC] via users
wrote:
> > From: Nico Kadel-Garcia
> > Sent: Saturday, May 11, 2024 10:47 AM
...
> > > I've tried with multiple repos of different sizes and ages. The
> > smaller repo I mentioned has about 150 files in trunk, mostly 50 KB or
> > smaller, and about 500 revisions. A larger repo with the same problems
> > has about 5000 files in trunk and 10,000 revisions.
> >
> > That *hints* at an httpd tuning issue, but I'm not sure. Check the httpd
> > logs?
>
> The httpd logs show no signs of a problem. Success and failure cases look
> the same in the logs.
In your initial post your mentioned:
>>> svn 1.10.2 was failing the same way before we upgraded to 1.14.1 as a
>>> possible fix.
So the problem isn't new after the upgrade.
BTW: as Daniel asked: You previously mentioned Subversion 1.14.1, is
that on the server or on the client?
Regardless, since the issue manifests on the client-side by hangs and
crashes while waiting for / processing data from the server (hangs
inside libsvn_ra_serf), I'd suggest also to investigate whether there
is something in between your client and your server that might be
interfering. A proxy or reverse proxy perhaps? Or some security
software on the client that interferes with the network (as some
antivirus suites do on Windows)? If so, try to bypass it (or disable /
create an exclude rule), as a diagnostic step to see whether this
might be the cause.
--
Johan
RE: [EXTERNAL] Re: svn checkout Hangs/Crashes/Succeeds Over HTTP
> From: Nico Kadel-Garcia > Sent: Saturday, May 11, 2024 10:47 AM > > > How did you upgrade your server from RHEL 6 to RHEL 8? > > > > Because so much changed from RHEL 6 to 8, including Apache from 2.2.15 > to 2.4.37, all the Apache modules, etc., I started from the skeleton > configuration the operating system provides and made mostly the same > customizations we had for RHEL 6, or modernized them where the docs said > things changed. Mostly, that was tweaks to authentication (from LDAP to > Kerberos), SSL, and the SVN endpoints. Browser access to all SVN and > ViewVC pages seems to work fine. > > Yeah. Forklift upgrades can be tricky. If I may suggest, segregate the > httpd running Subversion from one doing *anything* else. That was my plan too, to gradually strip off fluff and if necessary everything but SVN access to the repos. I'll report back what I find. > > I've tried with multiple repos of different sizes and ages. The > smaller repo I mentioned has about 150 files in trunk, mostly 50 KB or > smaller, and about 500 revisions. A larger repo with the same problems > has about 5000 files in trunk and 10,000 revisions. > > That *hints* at an httpd tuning issue, but I'm not sure. Check the httpd > logs? The httpd logs show no signs of a problem. Success and failure cases look the same in the logs. > > I thought svn:// worked only with svnserve, which we don't run. Are > you suggesting I try to run it as a test, or that I consider abandoning > Apache in favor of it? Yikes; that'd be painful. > > I'm suggesting you at least run it as a test, to validate that your > server's file systems and associated hardware are not messed up and > hiding some deeper issue. Switch only if you need to. If that's too > much work, you might try simply doing a file:/// based clone on the > server itself.. Good ideas. I'll try svnserve if my pared-down httpd configuration is inconclusive. The same checkouts using file:// URLs are consistently successful. That's not realistic for us, but it's still a useful data point, another indication the hangs are somehow in my server or network configuration. The core dumps are still an SVN client problem, but may need whatever that wonkiness is to be tickled. Jim
Re: [EXTERNAL] Re: svn checkout Hangs/Crashes/Succeeds Over HTTP
Hi,
I've added a few comments/questions below.
Kind regards,
Daniel Sahlberg
Den lör 11 maj 2024 kl 03:00 skrev Williams, James P. {Jim} (JSC-CD4)[KBR
Wyle Services, LLC] via users :
> > How did you upgrade your server from RHEL 6 to RHEL 8?
>
> Because so much changed from RHEL 6 to 8, including Apache from 2.2.15 to
> 2.4.37, all the Apache modules, etc., I started from the skeleton
> configuration the operating system provides and made mostly the same
> customizations we had for RHEL 6, or modernized them where the docs said
> things changed. Mostly, that was tweaks to authentication (from LDAP to
> Kerberos), SSL, and the SVN endpoints. Browser access to all SVN and
> ViewVC pages seems to work fine.
>
You previously mentioned Subversion 1.14.1, is that on the server or on the
client?
[...]
> And do the problems happen if you use svn:// rather than https:// ?
>
> I thought svn:// worked only with svnserve, which we don't run. Are you
> suggesting I try to run it as a test, or that I consider abandoning Apache
> in favor of it? Yikes; that'd be painful.
>
> I hear you on the HTTP integration. We have about 2000 repos and a few
> hundred developers. I've supported that server for at least 15 years, and
> it hasn't been too bad...until now.
>
I have personally only ever used Subversion over http/https (except for
testing purposes) and I haven't had any of the problems described by Nico -
I guess YMMV...
Still it would be interesting to compare just to rule out a problem within
the repository. You can use svnserve directly or tunneled over SSH, see the
Subversion book:
https://svnbook.red-bean.com/en/1.7/svn.serverconfig.svnserve.html#svn.serverconfig.svnserve.sshauth
> On Fri, May 10, 2024 at 4:17 PM Williams, James P. {Jim} (JSC-CD4)[KBR
> Wyle Services, LLC] via users wrote:
> >
> > I'm upgrading an Apache HTTP server of our SVN repos on RedHat
> Enterprise Linux 8. Using Subversion 1.14.1, svn checkout of even a small,
> simple repo with about 150 files hangs about 90% of the time, crashes 5%,
> and succeeds 5%. Given enough time, the hangs eventually time out after
> checking out much of the repo. A debugger shows the following stack during
> the hang.
> >
> >
> >
> > #0 epoll_wait /usr/lib64/libc.so.6
>
Waiting for a reply from the server ... ?
Do you see any activity on the server (CPU / disk) during this time?
> >
> > #1 impl_pollset_poll/usr/lib64/libapr-1.so.0
> >
> > #2 serf_context_run /usr/lib64/libserf-1.so.0
> >
> > #3 svn_ra_serf.context_run /usr/lib64/libsvn_ra_serf-1.so.0
> >
> > #4 finish_report/usr/lib64/libsvn_ra_serf-1.so.0
> >
> > #5 svn_wc_crawl_revisions5 /usr/lib64/libsvn_wc-1.so.0
> >
> > #6 update_internal.isra /usr/lib64/libsvn_client-1.so.0
> >
> > #7 svn_client.update_internal /usr/lib64/libsvn_client-1.so.0
> >
> > #8 svn_client.checkout_internal /usr/lib64/libsvn_client-1.so.0
> >
> > #9 svn_client_checkout3 /usr/lib64/libsvn_client-1.so.0
> >
> > #10 svn_cl.checkout
> >
> > #11 sub_main
> >
> > #12 main
> >
> >
> >
> > strace shows repeated calls to epoll_wait about 1 sec apart.
> >
> >
> >
> > When the checkout crashes, it's a SIGSEGV with this stack,
> >
> >
> >
> > #0 apr_pool_create_ex(libapr-1.so.0)
>
Memory allocation?
> >
> > #1 svn_pool_create_ex(libsvn_subr-1.so.0)
> >
> > #2 update_opened (libsvn_ra_serf-1.so.0)
> >
> > #3 expat_start (libsvn_ra_serf-1.so.0)
>
Parsing the XML message from the server?
Can you catch/view the actual XML message sent from the server? I'm
thinking if this is mangled in some strange way that is upsetting the XML
parser.
> >
> > #4 expat_start_handler (libsvn_subr-1.so.0)
> >
> > #5 doContent (libexpat.so.1)
> >
> > #6 contentProcessor (libexpat.so.1)
> >
> > #7 XML_ParseBuffer (libexpat.so.1)
> >
> > #8 svn_xml_parse (libsvn_subr-1.so.0)
> >
> > #9 expat_response_handler(libsvn_ra_serf-1.so.0)
> >
> > #10 process_buffer.isra.9 (libsvn_ra_serf-1.so.0)
> >
> > #11 finish_report (libsvn_ra_serf-1.so.0)
> >
> > #12 svn_wc_crawl_revisions5 (libsvn_wc-1.so.0)
> >
> > #13 update_internal.isra.0(libsvn_client-1.so.0)
> >
> > #14 svn_client__update_internal (libsvn_client-1.so.0)
> >
> > #15 svn_client__checkout_internal (libsvn_client-1.so.0)
> >
> > #16 svn_client_checkout3 (libsvn_client-1.so.0)
> >
> > #17 svn_cl__checkout (svn)
> >
> > #18 sub_main (svn)
> >
> > #19 main (svn)
> >
> > #20 __libc_start_main (libc.so.6)
> >
> > #21 _start
RE: [EXTERNAL] Re: svn checkout Hangs/Crashes/Succeeds Over HTTP
> How did you upgrade your server from RHEL 6 to RHEL 8?
Because so much changed from RHEL 6 to 8, including Apache from 2.2.15 to
2.4.37, all the Apache modules, etc., I started from the skeleton configuration
the operating system provides and made mostly the same customizations we had
for RHEL 6, or modernized them where the docs said things changed. Mostly,
that was tweaks to authentication (from LDAP to Kerberos), SSL, and the SVN
endpoints. Browser access to all SVN and ViewVC pages seems to work fine.
> Did you run "svnadmin upgrade" on the repo?
I haven't done anything to the repos. SVN release notes between 1.10 and 1.14
sounded like it was unnecessary, that a new server could operate on existing
repos. Regardless, I just ran "svnadmin upgrade" on these repos. Some were
bumped up to version 8, but it seems to have had no effect on the hangs and
crashes.
> And just how *big* is the repo?
I've tried with multiple repos of different sizes and ages. The smaller repo I
mentioned has about 150 files in trunk, mostly 50 KB or smaller, and about 500
revisions. A larger repo with the same problems has about 5000 files in trunk
and 10,000 revisions.
> Does your client setup work with a notably smaller repo?
Yeah, smaller repos are successful on most or all attempts. The revision count
seems to matter more than file count. One with 150 files and 500 revisions
fails almost every time. One with 225 files but only 42 revisions succeeds
almost every time. All of the files are small, mostly 50 KB or smaller.
> And do the problems happen if you use svn:// rather than https:// ?
I thought svn:// worked only with svnserve, which we don't run. Are you
suggesting I try to run it as a test, or that I consider abandoning Apache in
favor of it? Yikes; that'd be painful.
I hear you on the HTTP integration. We have about 2000 repos and a few hundred
developers. I've supported that server for at least 15 years, and it hasn't
been too bad...until now.
Thanks for the reply.
Jim
-Original Message-
From: Nico Kadel-Garcia
Sent: Friday, May 10, 2024 6:03 PM
To: Williams, James P. {Jim} (JSC-CD4)[KBR Wyle Services, LLC]
Cc: users@subversion.apache.org
Subject: [EXTERNAL] Re: svn checkout Hangs/Crashes/Succeeds Over HTTP
CAUTION: This email originated from outside of NASA. Please take care when
clicking links or opening attachments. Use the "Report Message" button to
report suspicious messages to the NASA SOC.
How did you upgrade your server from RHEL 6 to RHEL 8? Did you run
"svnadmin upgrade" on the repo? And just how *big* is the repo? Does
your client setup work with a notably smaller repo? And do the
problems happen if you use svn:// rather than https:// ? Because I've
been dealing with Subversion for a long time, and I have up on the
httpd integration years ago, The complex integration with an
overmuscled and constantly evolving, feature evolving webserver was
not something I bother with anymore, and it has big problems with big
repos, including its own repo over at http://svn.apache.org/repos/asf,
which can't be reliably synced anymore because it's gotten so large.
On Fri, May 10, 2024 at 4:17 PM Williams, James P. {Jim} (JSC-CD4)[KBR
Wyle Services, LLC] via users wrote:
>
> I'm upgrading an Apache HTTP server of our SVN repos on RedHat Enterprise
> Linux 8. Using Subversion 1.14.1, svn checkout of even a small, simple repo
> with about 150 files hangs about 90% of the time, crashes 5%, and succeeds
> 5%. Given enough time, the hangs eventually time out after checking out much
> of the repo. A debugger shows the following stack during the hang.
>
>
>
> #0 epoll_wait /usr/lib64/libc.so.6
>
> #1 impl_pollset_poll/usr/lib64/libapr-1.so.0
>
> #2 serf_context_run /usr/lib64/libserf-1.so.0
>
> #3 svn_ra_serf.context_run /usr/lib64/libsvn_ra_serf-1.so.0
>
> #4 finish_report/usr/lib64/libsvn_ra_serf-1.so.0
>
> #5 svn_wc_crawl_revisions5 /usr/lib64/libsvn_wc-1.so.0
>
> #6 update_internal.isra /usr/lib64/libsvn_client-1.so.0
>
> #7 svn_client.update_internal /usr/lib64/libsvn_client-1.so.0
>
> #8 svn_client.checkout_internal /usr/lib64/libsvn_client-1.so.0
>
> #9 svn_client_checkout3 /usr/lib64/libsvn_client-1.so.0
>
> #10 svn_cl.checkout
>
> #11 sub_main
>
> #12 main
>
>
>
> strace shows repeated calls to epoll_wait about 1 sec apart.
>
>
>
> When the checkout crashes, it's a SIGSEGV with this stack,
>
>
>
> #0 apr_pool_create_ex(libapr-1.so.0)
>
> #1 svn_pool_create_ex(libsvn_subr-1.so.0)
>
> #2 update_opened (libsvn_ra
