Re: [fossil-users] Standalone server slowing down significantly after long use
Thus said Philip Bennefall on Wed, 15 Jul 2015 20:22:07 +0200: About a month ago I reported having some issues with the standalone Fossil server, where it would slow down significantly after running for an extended period. What specifically is slow? Clone? Sync? Accessing pages served by the standalone Fossil server? All of these are network related, so what happens if you do the same things locally on the server to eliminate network slowness? For example, if clone is slow, login to the server and just clone locally: fossil clone http://server.blastbay.com/blastmidi/ /tmp/bmclone.fossil http://server.blastbay.com/blastmidi/ This one doesn't seem slow for me. I cloned it in 1 second. Browsing the site also seems fairly responsive. Now, I am in a position to gather any information that the developers might need. However I am new to Linux and have no idea what information might be of use, ... The things to look for are: How much memory the process is using. You can use ps for that. How much cpu is the process using. Again perhaps top. Has the process been reniced? If so, this means it will run with a lower priority than other processes and may respond slowly due to this. What journal mode do you have enabled on your Fossils? Also, since fossil server spawns a new process for each new connection, it might be hard to diagnose if the problem isn't necessarily with the server, but the forked server handler. Another thing to look at is how many file descriptors the process has open. With linux I believe you can use lsof to obtain this information (or /proc/PID/fd). You could use strace to profile where the process is going with syscalls and things. Might want to use -f to follow children since a new connection will fork. How much free memory does your system have? The free command should help with this. How about disk space? How about inodes? What about contention for I/O resources? E.g. is there something else hammering your disks which also slows down Fossil? What about netstat output? Specifically netstat -na? Does it show a lot of connections to your Fossil port either ESTABLISHED or in any other state? Maybe fossil is unable to allocate a new port because you're being DDoSed? Andy -- TAI64 timestamp: 400055a732a3 ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] Standalone server slowing down significantly after long use
Hi all, About a month ago I reported having some issues with the standalone Fossil server, where it would slow down significantly after running for an extended period. At that time I had just restarted the process, so was not able to gather any meaningful statistics in order to help diagnose the problem. Now, Fossil has been running on the server since that time and has become very slow again. An example of a repo on my server is: http://server.blastbay.com/blastmidi/ Now, I am in a position to gather any information that the developers might need. However I am new to Linux and have no idea what information might be of use, and even less what commands I should run to actually find that information. Would any of you Linux gurus be willing to give me some commands I could run on the fossil process to extract useful statistics? I am on the latest Ubuntu. Kind regards, Philip Bennefall On 6/2/2015 11:20 PM, Philip Bennefall wrote: Unfortunately I didn't check before I restarted the last time. I will keep running it, and gather statistics over time as it begins to slow down. Kind regards, Philip Bennefall On 6/2/2015 10:50 PM, Andy Bradford wrote: Thus said Philip Bennefall on Tue, 02 Jun 2015 20:12:15 +0200: This is a hard case to reproduce as it usually takes several weeks or even a month or two, but after the Fossil standalone server has been running for quite some time it becomes terribly sluggish. Has the process been reniced? How much memory does Fossil have before you restart? How many open file descriptors does it have before you restart? What journaling mode are you using? Thanks, Andy -- TAI64 timestamp: 4000556e1737 . ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users . ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Standalone server slowing down significantly after long use
On Jul 15, 2015, at 12:22 PM, Philip Bennefall phi...@blastbay.com wrote: Would any of you Linux gurus be willing to give me some commands I could run on the fossil process to extract useful statistics? Run top(1). Is fossil one of the top processes in its default mode, which sorts by CPU usage? If not, sort by memory usage instead.[*] If fossil is huge, your system is probably swapping, which would explain the slowdown. [*] The method varies by top(1) implementation. In some, pressing capital M works. In others, you need to re-launch it as top -m. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Standalone server slowing down significantly after long use
Some more thoughts: On Jul 15, 2015, at 12:22 PM, Philip Bennefall phi...@blastbay.com wrote: An example of a repo on my server is: http://server.blastbay.com/blastmidi/ What’s the largest repo on that server? That one is less than a meg and only has 3 checkins: http://server.blastbay.com/blastmidi/stat I doubt this one is the cause of your problems. Is anything else running on this server? General-purpose HTTP? Email? FTP? SSH for any purpose other than administration? I am new to Linux Who set this server up? Would you care to point us to the VPS or shared hosting plan you’re using? ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Standalone server slowing down significantly after long use
Hello Philip, On 15 July 2015 at 11:22, Philip Bennefall phi...@blastbay.com wrote: Now, Fossil has been running on the server since that time and has become very slow again. What _is_ slow? Loading the pages for me took about 0.004s. Your initial email reported taking 15 seconds or so to pull up the front page. Is this the problem? https://www.fossil-scm.org/index.html/doc/trunk/www/server.wiki You are running fossil server REPOSITORY and just keeping this in a tmux session or something? Is your ssh session (or any tcp connections) sluggish while your fossil server is also slow? -- --- inum: 883510009027723 sip: jungleboo...@sip2sip.info xmpp: jungle-boo...@jit.si ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Standalone server slowing down significantly after long use
Hi there, Thanks to all who responded! I decided to try running Fossil through xinetd instead, and am now seeing almost instantaneous load times again. The problem was never Fossil's actual processing time displayed at the bottom of each page (that was always just a few milliseconds), but rather the problem seemed to be related to the TCP communication. I configured the server myself. There are not many things running on it, but the few services that do run don't seem to be slowing down even when the standalone Fossil server is at a crawl. This includes other TCP connections - they also run fast all the time. It will be interesting to see how the server performs over time through Xinetd. Thanks again! Kind regards, Philip Bennefall P.S. I was running Fossil through a screen session before. On 7/15/2015 10:11 PM, jungle Boogie wrote: Hello Philip, On 15 July 2015 at 11:22, Philip Bennefall phi...@blastbay.com wrote: Now, Fossil has been running on the server since that time and has become very slow again. What _is_ slow? Loading the pages for me took about 0.004s. Your initial email reported taking 15 seconds or so to pull up the front page. Is this the problem? https://www.fossil-scm.org/index.html/doc/trunk/www/server.wiki You are running fossil server REPOSITORY and just keeping this in a tmux session or something? Is your ssh session (or any tcp connections) sluggish while your fossil server is also slow? ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Standalone server slowing down significantly after long use
On 6/2/15, Philip Bennefall phi...@blastbay.com wrote: after the Fossil standalone server has been running for quite some time it becomes terribly sluggish. ... I am running on a lower end Linode VPS, but so is the sqLite repository as far as I know and I have never had any trouble with that over the four years I've been following it. The difference is probably that the https://www.sqlite.org/src site is running fossil using CGI, not as a stand-alone server. Thanks for the report. We'll poke around and try to figure out what is going wrong. Which version of Fossil did you say you were using? And did you compile it yourself? -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] Standalone server slowing down significantly after long use
This is a hard case to reproduce as it usually takes several weeks or even a month or two, but after the Fossil standalone server has been running for quite some time it becomes terribly sluggish. You begin noticing a slower response times at first, and eventually it gets to a point where it is nearly unusable - taking 15 seconds or so to pull up the front page. As soon as I restart it, it is lightning fast again. Note that I do not restart the server machine itself - that's been running for a year or more, only the Fossil standalone server is restarted. Does anyone have any ideas why this might be happening? I am running on a lower end Linode VPS, but so is the sqLite repository as far as I know and I have never had any trouble with that over the four years I've been following it. Kind regards, Philip Bennefall ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Standalone server slowing down significantly after long use
On Jun 2, 2015, at 12:21 PM, Richard Hipp d...@sqlite.org wrote: On 6/2/15, Philip Bennefall phi...@blastbay.com wrote: after the Fossil standalone server has been running for quite some time it becomes terribly sluggish. ... I am running on a lower end Linode VPS, but so is the sqLite repository as far as I know and I have never had any trouble with that over the four years I've been following it. The difference is probably that the https://www.sqlite.org/src site is running fossil using CGI, not as a stand-alone server. You’re thinking some kind of memory leak, then? [*] That should be easy for the OP to [dis]prove, such as via top(1). For reference, my local instance is taking 0.0% of memory after running for a couple of weeks. (Meaning ~200 MiB, given that there is 6 GiB of physical RAM + 16 GiB of swap configured on this machine.) [*] I don’t mean to accuse SQLite or Fossil of completely losing track of allocated memory, but if it’s allocating chunks of memory that don’t get properly freed until the program exits, it amounts to the same thing. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Standalone server slowing down significantly after long use
Thus said Philip Bennefall on Tue, 02 Jun 2015 20:12:15 +0200: This is a hard case to reproduce as it usually takes several weeks or even a month or two, but after the Fossil standalone server has been running for quite some time it becomes terribly sluggish. Has the process been reniced? How much memory does Fossil have before you restart? How many open file descriptors does it have before you restart? What journaling mode are you using? Thanks, Andy -- TAI64 timestamp: 4000556e1737 ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Standalone server slowing down significantly after long use
Unfortunately I didn't check before I restarted the last time. I will keep running it, and gather statistics over time as it begins to slow down. Kind regards, Philip Bennefall On 6/2/2015 10:50 PM, Andy Bradford wrote: Thus said Philip Bennefall on Tue, 02 Jun 2015 20:12:15 +0200: This is a hard case to reproduce as it usually takes several weeks or even a month or two, but after the Fossil standalone server has been running for quite some time it becomes terribly sluggish. Has the process been reniced? How much memory does Fossil have before you restart? How many open file descriptors does it have before you restart? What journaling mode are you using? Thanks, Andy -- TAI64 timestamp: 4000556e1737 . ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
