Re: [Savannah-hackers-public] VERY slow git response from Savannah
arn...@skeeve.com wrote: > I would have thought most people would do 'git pull' instead of 'git clone' > and that pulling wouldn't be quite as intensive, but who knows... I think that these days most people do not keep persistent state. The cynic in me assumes they are working from their phone. Instead they clone a new repository, do something, then toss it away. Even for continuous integration systems many people do not use a local cache. I can't prove this but it is just my observation of the way people work around me in real life. This makes large projects very I/O intensive when things happen. I was actually using clone generically there. But if people were pulling then I would have expected the processes to have finished quickly. But we do periodically see large repositories getting cloned at the same time due to project announcements all run at the same time and take up time. I don't know which projects were getting pulled or cloned since we do not log that information. But previously Emacs has been the cause of it becaues the repos is large (and originally was larger) making cloning need a lot of bandwidth. When Emacs announces a new release there is usually a spike. So using it as an example without saying that was the project this time. Bob
Re: [Savannah-hackers-public] VERY slow git response from Savannah
On Wed, Feb 21, 2018 at 12:45:24PM -0700, Bob Proulx wrote: > arn...@skeeve.com wrote: > > I would have thought most people would do 'git pull' instead of 'git clone' > > and that pulling wouldn't be quite as intensive, but who knows... > > I think that these days most people do not keep persistent state. The > cynic in me assumes they are working from their phone. Instead they > clone a new repository, do something, then toss it away. Even for > continuous integration systems many people do not use a local cache. > I can't prove this but it is just my observation of the way people > work around me in real life. This makes large projects very I/O > intensive when things happen. As a relative young'un with a smartphone, I can say that doing development on my phone sounds like a bad time! I typically keep full clones around, even for one-off things. Source code is small and gigabytes are cheap. > I was actually using clone generically there. But if people were > pulling then I would have expected the processes to have finished > quickly. But we do periodically see large repositories getting cloned > at the same time due to project announcements all run at the same time > and take up time. I don't know which projects were getting pulled or > cloned since we do not log that information. But previously Emacs has > been the cause of it becaues the repos is large (and originally was > larger) making cloning need a lot of bandwidth. When Emacs announces > a new release there is usually a spike. So using it as an example > without saying that was the project this time. I imagine there are CI systems that do full clones and then check out the commit they want. Of course this is a big waste of bandwidth, so in Guix we are looking into the possibility of shallow cloning for related use cases. This may be less efficient on the server; I guess it depends on the server implementation. Still, we favor release tarballs because they are small and the required software is much less complex. signature.asc Description: PGP signature
Re: [Savannah-hackers-public] Is something wrong with Savannah's Git server?
> Date: Wed, 21 Feb 2018 02:45:27 -0700 > From: Bob Proulx > Cc: savannah-hackers-public@gnu.org > > Eventually I couldn't find an alternative to killing all of the git > processes and waiting a bit before restarting the server. Hopefully > as people start cloning again they won't all do it at once. A little > time skew goes a long way. > > Having done that at around 06:20 UTC (11:20pm US/Mountain) that seemed > to break the problem and things appear to be working normally again > since then. I have once again reduced the number of parallel git > servers, now down from 25 to 15. Thanks, it seems to be working OK now.
Re: [Savannah-hackers-public] VERY slow git response from Savannah
Hi Bob, See below. Bob Proulx wrote: > Arnold Robbins wrote: > > I'm seeing terribly slow git response from Savannah, both in Israel > > and from a machine on the US east coast. So I think the problem isn't > > network related. > ... > > If anyone is awake (I know it's about 1:00 a.m. east coast time), please > > investigate. > > I believe that one of the projects hosted must have announced a new > git release and that synchronized a lot of people trying to clone at > the same time. It seemed as if everyone in the world was trying to > clone all at once. There was almost 8G of system memory consumed > solely by many running git processes. It just couldn't keep up. > > Eventually I couldn't find an alternative to killing all of the git > processes and waiting a bit before restarting the git daemon. > Hopefully as people start cloning again they won't all do it at once. > A little time skew goes a long way. > > Having done that at around 06:20 UTC (11:20pm US/Mountain) that seemed > to break the problem and things appear to be working normally again > since then. I have once again reduced the number of parallel git > servers, now down from 25 to 15. > > Bob Much thanks. That seems to have done the trick; it's behaving normally for me now. I would have thought most people would do 'git pull' instead of 'git clone' and that pulling wouldn't be quite as intensive, but who knows... Thanks again - now go get some sleep. :-) Arnold
Re: [Savannah-hackers-public] VERY slow git response from Savannah
Arnold Robbins wrote: > I'm seeing terribly slow git response from Savannah, both in Israel > and from a machine on the US east coast. So I think the problem isn't > network related. ... > If anyone is awake (I know it's about 1:00 a.m. east coast time), please > investigate. I believe that one of the projects hosted must have announced a new git release and that synchronized a lot of people trying to clone at the same time. It seemed as if everyone in the world was trying to clone all at once. There was almost 8G of system memory consumed solely by many running git processes. It just couldn't keep up. Eventually I couldn't find an alternative to killing all of the git processes and waiting a bit before restarting the git daemon. Hopefully as people start cloning again they won't all do it at once. A little time skew goes a long way. Having done that at around 06:20 UTC (11:20pm US/Mountain) that seemed to break the problem and things appear to be working normally again since then. I have once again reduced the number of parallel git servers, now down from 25 to 15. Bob
Re: [Savannah-hackers-public] Is something wrong with Savannah's Git server?
Eli Zaretskii wrote: > It is extremely unresponsive lately. I type "git pull" and wait for > several minutes(!) with nothing happening. > > Help! I believe that one of the projects hosted must have announced a new git release and that synchronice a lot of people trying to clone at the same time. It seemed as if everyone in the world was trying to clone all at once. There was almost 8G of system memory consumed solely by many running git processes. It just couldn't keep up. Eventually I couldn't find an alternative to killing all of the git processes and waiting a bit before restarting the server. Hopefully as people start cloning again they won't all do it at once. A little time skew goes a long way. Having done that at around 06:20 UTC (11:20pm US/Mountain) that seemed to break the problem and things appear to be working normally again since then. I have once again reduced the number of parallel git servers, now down from 25 to 15. Bob
