Re: [fossil-users] Stress testing www4... (was Re: [Bug?] [server] Processes of Fossil popping up unexpectedly
On 12/22/17, John P. Rouillardwrote: > > Well since reverse proxy servers use a client idle setting, would I be > crazy to suggest the same for fossil? > I did a half-baked idle timer in the latest check-in, using the --max-latency option. fossil server --max-latency 30 The above will automatically terminate each request after 30 seconds, finished or not. I started out trying to implement a more complex idle timeout, but that was a lot of code to test, and it is late here. Maybe we can add something like that later. Or maybe we can tell people to use a reverse proxy if they are deploying in an environment where a more advanced idle timeout is necessary to keep bad actors at bay. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Stress testing www4... (was Re: [Bug?] [server] Processes of Fossil popping up unexpectedly
Hi Richard: In message
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
The latest trunk check-in on Fossil (http://www4.fossil-scm.org/info/05ec15cad53e8176) should do a much better job of handling load from "fossil server". Please compile and run it and let me know what happens. The www4.fossil-scm.org server is running as follows: /usr/local/bin/fossil serve -port 80 -nojail -max-latency 30 /root/Fossils/fossil.fossil That server is one of the €2.99/month servers available from www.scaleway.com - an ARM64 with 2GB of RAM located in the Paris datacenter. I'll leave it running for a few days, but I do not intend for it to be a permanent mirror, so the link to www4.fossil-scm.org above may be dead if you are delayed in reading this posting. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Stress testing www4... (was Re: [Bug?] [server] Processes of Fossil popping up unexpectedly
On 12/22/17, John P. Rouillardwrote: > > I am currently running 5 > > nc www4.fossil-scm.org 80 > > processes and sending no input. Ah hah! I think the problem is this code: http://www4.fossil-scm.org/artifact/9694a7dc?ln=1855-1858 That connection rate throttler is code that predates Fossil. I copied it from CVSTrac. It is present in the very first check-in of Fossil (2007-07-21). MAX_PARALLEL is currently set to 2, which is a ridiculously low number. I have raised it to 50 on www4.fossil-scm.org and restarted that server. I should probably raise it to 1000 or so. And instead of sleeping, I'll just block until children die off. This problem only comes up when running "fossil server" with direct connections to the outside, and rogue outsiders starting up connections and just sitting on them. People using a reverse proxy (ex: nginx) won't see the problem because the reverse proxy never sits on a TCP/IP connection. Give me a few minutes to come up with the best solution. Suggestions from the mailing list are also welcomed. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
In message, Warren Young writes: >On Dec 22, 2017, at 10:20 AM, Olivier R. wrote: >> >> Le 22/12/2017 à 16:10, Warren Young a écrit : >>> 1. Your repo is public-facing. Is this a reasonable number of >>> clients to be connected at any given time to this repo? It seems >>> high to me, given the transient nature of most Fossil connections. >> >> Only two devs have the right to modify the online repository. >> I’d be surprised if there were more than 5 people connected to this >> repo at the same time. > >Your first netstat -na output shows 24 established connections, most >to distinct remote peer IPs. > >Additionally, Fossil connections tend to be short-lived due to the >nature of Fossil. [...] >Therefore, one of two things is happening: > >1. You have a high rate of connections to the server, so that at any > one time, we can see dozens of them. Disproving this hypothesis is > the purpose of the tshark test. > >2. You have connections that stay open for a long time, which suggests: > >2a. A connection handle leak. But if this were common, lots of >people would have found it before you. > >2b. Bad actors on those remote hosts, which is why I bought up the >possibility of a botnet attack. > >If we have a 2b case, then you’d want to find some way to identify >the connections somehow, so that you can block them. The only way I managed to reproduce the lsof output that was reported earlier was to connect to my fossil using netcat and send no data. Once I sent any data (by hitting a return) fossil reported a 403 (since I sent no http headers). As I noted earlier I have hiawatha running in front of the fossil server. It kills idle connections after I think 60 seconds. So if there is no traffic, the connection to the back end fossil server will be killed. Note that Warren is running nginx as the front end for his fossil server. I think that has anti-DOS and timeouts on client header/body and response times that can do something similar. Does fossil have any idle connection timeout limits? -- -- rouilj John Rouillard === My employers don't acknowledge my existence much less my opinions. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
Le 22/12/2017 à 21:34, Richard Hipp a écrit : On 12/22/17, Olivier R.wrote: I also run Fossil on a cheap VPS. https://www.scaleway.com/virtual-cloud-servers/ (The starter version at €2.99/month) I'm building up a new Fossil server on this VPS now - on an ARM64-2G machine in Paris. It is slow-going. Apparently the 2.99/mo ARM64 machines are not very fast :-) I’m not using the ARM version but the x86-64 version. And it’s fast enough for me and fits my needs. My repo is small: only 130 Gb. Note that the bug usually happens after weeks or months, but then it slows down quite quickly as new processes seem more likely to pop up when it has begun. Olivier ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
Le 22/12/2017 à 22:39, Richard Hipp a écrit : On 12/22/17, Olivier R.wrote: I also run Fossil on a cheap VPS. Maybe you have used up your bandwidth quota and the ISP is throttling your connection? There is no bandwidth quota. And when I kill the processes and relaunch Fossil, it runs quickly. Olivier ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On 12/22/17, Olivier R.wrote: > > I also run Fossil on a cheap VPS. Maybe you have used up your bandwidth quota and the ISP is throttling your connection? -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Fossil hack: quickly finding the first checkin via the web UI
On Thu 21 Dec 2017 11:46 AM, Dominique Devienne wrote: > > Nice tip. > > Since I've seen DRH ask about available features on GitHub since I started > lurking on this list recently, > I'd mention the graphs the latter provide which I find useful, in the > Insights tab of a project. > The Code Frequency [1] one seems to be since inception, while the Commit > [2] one is year-to-date. > I also like the per-user "map" that shows activity of a GitHub user, across > all projects, but there's > also a per-project page with contributor graphs [3], also since inception > it seems. > > I find such graphs useful myself, FWIW. --DD > > [1] https://github.com/rqlite/rqlite/graphs/code-frequency > [2] https://github.com/rqlite/rqlite/graphs/commit-activity > [3] https://github.com/rqlite/rqlite/graphs/contributors Are you familiar with these: http://www4.fossil-scm.org/stat http://www4.fossil-scm.org/reports http://www4.fossil-scm.org/reports?type=ci=bymonth=drh http://www4.fossil-scm.org/reports?type=g=byfile=drh Play around with these and other features. My guess is you'll find out how much is reported. There could be a chance to add more and/or improve on what is in place, however. P.s. I'm using www4 because of DRH's request on the other thread. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Digital signatures on check-ins. Was: tangent vs. wyoung on recent commti
On Fri 22 Dec 2017 6:13 AM, Warren Young wrote: > On Dec 21, 2017, at 4:57 PM, jungle Boogiewrote: > > > > On 21 December 2017 at 15:03, Warren Young wrote: > >> On Dec 21, 2017, at 2:58 PM, jungle Boogie wrote: > >>> perhaps left of the username. > >> > >> I think right, simply because it’s currently ordered most-clicked to > >> least-clicked, more or less. > >> > > > > In between the username and date/time or right of date/time? > > I was thinking far right, after the tag in the Modern view. I sometimes > click on the checkin ID, I almost never click on the user name, and when I > click on a tag, it’s generally from some other view than the Timeline. Oh, I was thinking in the overview section of a commit, left of the username: http://www4.fossil-scm.org/info/8a53d4016ee960ab Shift what's there down a bit to have the icon line up with the other records above and below. With what you said, I do think that placement is nice. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On 12/22/17, Richard Hippwrote: > On 12/22/17, Olivier R. wrote: >> >> I also run Fossil on a cheap VPS. >> https://www.scaleway.com/virtual-cloud-servers/ >> (The starter version at €2.99/month) >> > > I'm building up a new Fossil server on this VPS now The new server is now up and running. Please visit http://www4.fossil-scm.org/ I compiled the latest trunk version using "./configure --with-openssl=none --no-opt". I'll try to keep an eye on this system to see how much CPU and memory it is using. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On 12/22/17, Olivier R.wrote: > > I also run Fossil on a cheap VPS. > https://www.scaleway.com/virtual-cloud-servers/ > (The starter version at €2.99/month) > I'm building up a new Fossil server on this VPS now - on an ARM64-2G machine in Paris. It is slow-going. Apparently the 2.99/mo ARM64 machines are not very fast :-) Assuming I get this running, I'll start up a "fossil server" and let people pound on it, in order to try to repro the problem. I'll let you know once the build finishes -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
Le 22/12/2017 à 18:41, Warren Young a écrit : I’ve added the user to the “wireshark” group, but it doesn’t work. You have to log out and back in before group changes will take effect. That’s what I did. But it didn’t work. So I used `lsof -i:8080` again. Here is the difference with the previous connections: https://www.diffchecker.com/KRGBfoK1 So it seems that almost all connections seen few hours ago are still there. Olivier ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
Le 22/12/2017 à 19:57, Warren Young a écrit : On Dec 22, 2017, at 9:03 AM, Warren Youngwrote: Olivier, what about memory usage for the Fossil processes? “top” can give you this. I just checked my public server, and each of the 4 Fossil instances is taking about 20 MB of VM, of which only about 800 kB is resident. top calculates this as 0.2% of the server’s small allocation of virtual RAM. (It’s a cheap VPS, not a real server.) I also run Fossil on a cheap VPS. https://www.scaleway.com/virtual-cloud-servers/ (The starter version at €2.99/month) There is no memory overload (it seems). top - 19:40:57 up 238 days, 9:50, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 110 total, 1 running, 108 sleeping, 0 stopped, 1 zombie %Cpu(s): 0.2 us, 0.0 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.5 st KiB Mem: 2049824 total, 1782024 used, 267800 free, 155636 buffers KiB Swap:0 total,0 used,0 free. 1436172 cached Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 10468 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 10469 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 10470 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 10471 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 10474 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 10475 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 10477 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 12333 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 12334 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 12335 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 12336 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 12337 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 12340 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 12341 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 fossil2 15143 myuser20 0 14876 3452 2724 S 0.0 0.2 0:00.00 fossil2 15144 myuser20 0 14876 3452 2724 S 0.0 0.2 0:00.00 fossil2 17096 myuser20 0 80656 4420 3568 S 0.0 0.2 0:00.01 sshd 17097 myuser20 0 12708 1760 1600 S 0.0 0.1 0:00.00 sftp-server 17098 myuser20 0 22784 4640 3340 S 0.0 0.2 0:00.09 bash 17107 myuser20 0 0 0 0 Z 0.0 0.0 0:00.01 fossil2 17108 myuser20 0 23628 2896 2440 R 0.0 0.1 0:00.04 top 17876 myuser20 0 14872 4248 3524 S 0.0 0.2 0:41.50 fossil2 20253 myuser20 0 14876 3452 2724 S 0.0 0.2 0:00.00 fossil2 25581 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 fossil2 25582 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 fossil2 28558 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 fossil2 28559 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 fossil2 30076 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 fossil2 30077 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 fossil2 31100 myuser20 0 14876 3452 2724 S 0.0 0.2 0:00.00 fossil2 Olivier ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On Dec 22, 2017, at 9:03 AM, Warren Youngwrote: > > Olivier, what about memory usage for the Fossil processes? “top” can give > you this. I just checked my public server, and each of the 4 Fossil instances is taking about 20 MB of VM, of which only about 800 kB is resident. top calculates this as 0.2% of the server’s small allocation of virtual RAM. (It’s a cheap VPS, not a real server.) I’ve taken the opportunity to upgrade, which changes VM not a whit; RSS went up to ~1.7 MB, doubtless to be whittled down over time by the VMM; and /timeline takes 8 ms now. So no, my server hadn’t been slowed down at all in that 11 week 4 day span. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On Dec 22, 2017, at 11:31 AM, Warren Youngwrote: > > I posted a long HOWTO about it back in June of 2016: > > https://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg23320.html I just realized that that message is a reply to my post, not the post itself. Here’s the direct link: https://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg22907.html It’s a fair bit of work, but it’s worth it for TLS protection of login credentials alone. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On Dec 22, 2017, at 10:53 AM, Tomek Kottwrote: > > FWIW I also run in server mode constantly because its easy. My users (there > are 10, they probably hit the server in bursts of 10 times one one day of the > week) occasionally notice slowdowns and ping me. When I said above that I occasionally restart mine, I meant only for upgrades to newer versions of Fossils. I can’t ever remember restarting it because it was slow. My public Fossil server was last restarted on October 2. I just pulled up a timeline in 7ms. My public server has 4 Fossil instances, one for each repo it serves. (Not important why.) Each instance is bound to localhost, with an nginx proxy in front of them. I’ve just checked, and only those 4 processes are currently running, no children. They’re running in SCGI mode, for whatever that’s worth. Olivier, if you want to try my configuration, I posted a long HOWTO about it back in June of 2016: https://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg23320.html It’s obsolete in places because nginx + Let’s Encrypt certificates no longer require manual updates. I won’t be motivated to rewrite it until I move to another hosting provider, which will force me to rebuild my site configuration. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
FWIW I also run in server mode constantly because its easy. My users (there are 10, they probably hit the server in bursts of 10 times one one day of the week) occasionally notice slowdowns and ping me. This is for an internal site only, no public facing, so shouldn't be huge numbers of users. I restart the server and all is well. I don't see this as a problem since I know I should be running a 'real' server, but *shrug* this is good enough and a simple restart works well. Tomek From: fossil-userson behalf of fossil-users-requ...@lists.fossil-scm.org Sent: Friday, December 22, 2017 12:41 PM To: fossil-users@lists.fossil-scm.org Subject: fossil-users Digest, Vol 119, Issue 49 In message
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On Dec 22, 2017, at 10:20 AM, Olivier R.wrote: > > Le 22/12/2017 à 16:10, Warren Young a écrit : > >> 1. Your repo is public-facing. Is this a reasonable number of >> clients to be connected at any given time to this repo? It seems >> high to me, given the transient nature of most Fossil connections. > > Only two devs have the right to modify the online repository. > I’d be surprised if there were more than 5 people connected to this repo at > the same time. Your first netstat -na output shows 24 established connections, most to distinct remote peer IPs. Additionally, Fossil connections tend to be short-lived due to the nature of Fossil. Even if you have someone “actively” working with Fossil, the connections will come and go rapidly in a series, not have a single long-lived connection. Therefore, one of two things is happening: 1. You have a high rate of connections to the server, so that at any one time, we can see dozens of them. Disproving this hypothesis is the purpose of the tshark test. 2. You have connections that stay open for a long time, which suggests: 2a. A connection handle leak. But if this were common, lots of people would have found it before you. 2b. Bad actors on those remote hosts, which is why I bought up the possibility of a botnet attack. If we have a 2b case, then you’d want to find some way to identify the connections somehow, so that you can block them. > I assume that people who go to dev section are not the majority. Maybe 20 > visitors per day if I’m optimistic… 20 *legitimate* visitors. That doesn’t count those hitting your site for no good reason, which does happen on the wild Internet. > I’ve added the user to the “wireshark” group, but it doesn’t work. You have to log out and back in before group changes will take effect. > Same error message if i run this as root. Better to follow tshark’s advice and run it as a normal user, with group privileges allowing network capture. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
Le 22/12/2017 à 16:10, Warren Young a écrit : 1. Your repo is public-facing. Is this a reasonable number of clients to be connected at any given time to this repo? It seems high to me, given the transient nature of most Fossil connections. Only two devs have the right to modify the online repository. I’d be surprised if there were more than 5 people connected to this repo at the same time. lsof -i Here again I expected you to look at the docs and infer that I meant for you to filter out only the interesting ports, 8080 in your case. -i:8080. Hmm. Sorry. As I said before, this server only runs Fossil and has no other purpose. lsof -i:8080 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME fossil2 10468 myuser0u IPv4 42925675 0t0 TCP scw-f45a9f:http-alt->hn.kd.ny.adsl:59187 (ESTABLISHED) fossil2 10468 myuser1u IPv4 42925675 0t0 TCP scw-f45a9f:http-alt->hn.kd.ny.adsl:59187 (ESTABLISHED) fossil2 10468 myuser2u IPv4 42925675 0t0 TCP scw-f45a9f:http-alt->hn.kd.ny.adsl:59187 (ESTABLISHED) fossil2 10468 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) fossil2 10469 myuser0u IPv4 42925676 0t0 TCP scw-f45a9f:http-alt->112.80.138.198:54839 (ESTABLISHED) fossil2 10469 myuser1u IPv4 42925676 0t0 TCP scw-f45a9f:http-alt->112.80.138.198:54839 (ESTABLISHED) fossil2 10469 myuser2u IPv4 42925676 0t0 TCP scw-f45a9f:http-alt->112.80.138.198:54839 (ESTABLISHED) fossil2 10469 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) fossil2 10470 myuser0u IPv4 42925678 0t0 TCP scw-f45a9f:http-alt->182.138.158.40:36008 (ESTABLISHED) fossil2 10470 myuser1u IPv4 42925678 0t0 TCP scw-f45a9f:http-alt->182.138.158.40:36008 (ESTABLISHED) fossil2 10470 myuser2u IPv4 42925678 0t0 TCP scw-f45a9f:http-alt->182.138.158.40:36008 (ESTABLISHED) fossil2 10470 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) fossil2 10471 myuser0u IPv4 42925679 0t0 TCP scw-f45a9f:http-alt->112.64.209.135:44966 (ESTABLISHED) fossil2 10471 myuser1u IPv4 42925679 0t0 TCP scw-f45a9f:http-alt->112.64.209.135:44966 (ESTABLISHED) fossil2 10471 myuser2u IPv4 42925679 0t0 TCP scw-f45a9f:http-alt->112.64.209.135:44966 (ESTABLISHED) fossil2 10471 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) fossil2 10474 myuser0u IPv4 42925688 0t0 TCP scw-f45a9f:http-alt->175.42.2.225:51516 (ESTABLISHED) fossil2 10474 myuser1u IPv4 42925688 0t0 TCP scw-f45a9f:http-alt->175.42.2.225:51516 (ESTABLISHED) fossil2 10474 myuser2u IPv4 42925688 0t0 TCP scw-f45a9f:http-alt->175.42.2.225:51516 (ESTABLISHED) fossil2 10474 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) fossil2 10475 myuser0u IPv4 42925700 0t0 TCP scw-f45a9f:http-alt->124.235.138.25:64841 (ESTABLISHED) fossil2 10475 myuser1u IPv4 42925700 0t0 TCP scw-f45a9f:http-alt->124.235.138.25:64841 (ESTABLISHED) fossil2 10475 myuser2u IPv4 42925700 0t0 TCP scw-f45a9f:http-alt->124.235.138.25:64841 (ESTABLISHED) fossil2 10475 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) fossil2 10477 myuser0u IPv4 42925703 0t0 TCP scw-f45a9f:http-alt->221.13.12.222:34854 (ESTABLISHED) fossil2 10477 myuser1u IPv4 42925703 0t0 TCP scw-f45a9f:http-alt->221.13.12.222:34854 (ESTABLISHED) fossil2 10477 myuser2u IPv4 42925703 0t0 TCP scw-f45a9f:http-alt->221.13.12.222:34854 (ESTABLISHED) fossil2 10477 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) fossil2 12333 myuser0u IPv4 42952323 0t0 TCP scw-f45a9f:http-alt->221.13.12.209:50846 (ESTABLISHED) fossil2 12333 myuser1u IPv4 42952323 0t0 TCP scw-f45a9f:http-alt->221.13.12.209:50846 (ESTABLISHED) fossil2 12333 myuser2u IPv4 42952323 0t0 TCP scw-f45a9f:http-alt->221.13.12.209:50846 (ESTABLISHED) fossil2 12333 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) fossil2 12334 myuser0u IPv4 42952324 0t0 TCP scw-f45a9f:http-alt->182.138.162.53:64088 (ESTABLISHED) fossil2 12334 myuser1u IPv4 42952324 0t0 TCP scw-f45a9f:http-alt->182.138.162.53:64088 (ESTABLISHED) fossil2 12334 myuser2u IPv4 42952324 0t0 TCP scw-f45a9f:http-alt->182.138.162.53:64088 (ESTABLISHED) fossil2 12334 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) fossil2 12335 myuser0u IPv4 42952325 0t0 TCP scw-f45a9f:http-alt->106.114.63.0:17736 (ESTABLISHED) fossil2 12335 myuser1u IPv4 42952325 0t0 TCP scw-f45a9f:http-alt->106.114.63.0:17736 (ESTABLISHED) fossil2 12335 myuser2u IPv4 42952325 0t0 TCP scw-f45a9f:http-alt->106.114.63.0:17736 (ESTABLISHED) fossil2 12335 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) fossil2 12336 myuser0u IPv4 42952329 0t0 TCP scw-f45a9f:http-alt->139.227.182.157:55348 (ESTABLISHED) fossil2 12336 myuser
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
Hello all: In message
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On Dec 22, 2017, at 8:56 AM, Richard Hippwrote: > > (afaik) nobody uses "fossil server" > for long-running websites. I do. Mine stay running for months at a time, including public-facing ones. > use the xinetd/inetd style of invoking Fossil I can only see that helping if there is some kind of leak. We can see from his test results that there is no socket descriptor leak. Olivier, what about memory usage for the Fossil processes? “top” can give you this. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On 12/22/17, Olivier R.wrote: > > I didn’t use Fossil 2.3 because the SQLite version bundled was in beta > stage. That is NOT a good reason to reject the use of Fossil 2.3. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On 12/22/17, Olivier R.wrote: > > I’ll have to kill all these processes soon and relaunch Fossil. > I’ll do it when you think you have enough information about this > problem. I’ll use Fossil 2.4 next time. > Maybe there is some kind of issue in the HTTP server implementation in cgi.c that causes it to slow down after running for a long time. This might have gone unnoticed, since (afaik) nobody uses "fossil server" for long-running websites. I will investigate and try to repro the problem when I have time. In the meantime, please consider setting up your website to use the xinetd/inetd style of invoking Fossil, as described in the https://www.fossil-scm.org/fossil/doc/trunk/www/server.wiki document. This is what Fossil itself uses, and also SQLite. It has been running for years under heavy load without issues. We know that method works well. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On Dec 22, 2017, at 8:24 AM, Olivier R.wrote: > > I’ll have to kill all these processes soon and relaunch Fossil. > I’ll do it when you think you have enough information about this problem. My request for connection rates doesn’t require that you keep Fossil running — in fact it would give correct results even if Fossil is down! — but drh may require state you can only learn from the executable still running. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
Le 22/12/2017 à 15:59, Richard Hipp a écrit : On 12/22/17, Olivier R.wrote: There is now more than 24 subprocesses of Fossil running, and it’s getting really slow. Are you saying that these 24 Fossils are spinning - using CPU cycles - not just hung waiting on I/O? These processes doesn’t seem to use much CPU time, but actually interaction with Fossil is now *very* slow. top - 15:19:22 up 238 days, 5:28, 1 user, load average: 0.00, 0.00, 0.00 Tasks: 111 total, 1 running, 109 sleeping, 0 stopped, 1 zombie %Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 99.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.3 st KiB Mem: 2049824 total, 1650376 used, 399448 free, 154816 buffers KiB Swap:0 total,0 used,0 free. 1308056 cached Mem PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 12873 myuser20 0 80660 4540 3696 S 0.0 0.2 0:00.11 `- sshd 13229 myuser20 0 80660 4456 3608 S 0.0 0.2 0:00.02 `- sshd 13230 myuser20 0 12708 1780 1624 S 0.0 0.1 0:00.00 `- sftp-server 13231 myuser20 0 22784 4564 3268 S 0.0 0.2 0:00.09 `- bash 13241 myuser20 0 23628 3012 2456 R 0.3 0.1 0:00.38 `- top 17876 myuser20 0 14872 4248 3524 S 0.0 0.2 0:41.40 `- fossil2 31100 myuser20 0 14876 3452 2724 S 0.0 0.2 0:00.00 `- fossil2 30076 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 `- fossil2 30077 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 `- fossil2 20253 myuser20 0 14876 3452 2724 S 0.0 0.2 0:00.00 `- fossil2 25581 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 `- fossil2 25582 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 `- fossil2 15143 myuser20 0 14876 3452 2724 S 0.0 0.2 0:00.00 `- fossil2 15144 myuser20 0 14876 3452 2724 S 0.0 0.2 0:00.00 `- fossil2 28558 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 `- fossil2 28559 myuser20 0 14876 3448 2720 S 0.0 0.2 0:00.00 `- fossil2 10468 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 10469 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 10470 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 10471 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 10474 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 10475 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 10477 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 12333 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 12334 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 12335 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 12336 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 12337 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 12340 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 12341 myuser20 0 14876 3516 2784 S 0.0 0.2 0:00.00 `- fossil2 13247 myuser20 0 0 0 0 Z 0.0 0.0 0:00.01 `- fossil2 You can see how slow it is by going there: http://212.47.254.152:8080 In normal times, it works fast. I’ll have to kill all these processes soon and relaunch Fossil. I’ll do it when you think you have enough information about this problem. I’ll use Fossil 2.4 next time. Regards, Olivier ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On Dec 22, 2017, at 8:10 AM, Warren Youngwrote: > > $ sudo tshark -b duration:1 port 8080 and tcp.flags.syn==1 | wc -l Sorry, that’s bogus. Try this instead: $ sudo tshark -i enp3s0 -w x.pcap -b duration:5 -a files:1 \ port 8080 and "tcp[tcpflags] & tcp-syn != 0” Adjust the interface name (-i) to suit your machine. The capture file x.pcap is just there to placate tshark. You can remove it when the command finishes, because what we’re interested in is the number that tshark reports. That’s also why we don’t need wc -l. And as for the SYN filter, that was a display filter, not a capture filter. Sigh. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On Dec 22, 2017, at 7:46 AM, Olivier R.wrote: > > Le 19/12/2017 à 21:49, Warren Young a écrit : > >> If it’s a sign of a bug, then it means something very bad has >> happened, like the network stack has lost track of its client >> somehow. To see that, you’d need to do a network capture on that >> fossil instance’s network sockets. Use netstat -nap or lsof -i to >> find out which TCP ports those are. > > There is now more than 24 subprocesses of Fossil running, and it’s getting > really slow. > > netstat -na | grep :8080 > > tcp0 0 0.0.0.0:80800.0.0.0:* LISTEN > tcp0 0 10.2.148.67:8080125.46.156.43:59187 ESTABLISHED > tcp0 0 10.2.148.67:8080175.42.2.225:51516 ESTABLISHED Two things I learn from this: 1. Your repo is public-facing. Is this a reasonable number of clients to be connected at any given time to this repo? It seems high to me, given the transient nature of most Fossil connections. 2. I don’t see many WAIT states, so we’re probably not looking at the sort of weird bug as in that Cloudflare blog post I linked to earlier. > After clicking on “timeline” on the web ui. > > netstat -nap | grep :8080 The -p in that command was intended to let you find the Fossil process by name, rather than by port. So, “netstat -nap | grep fossil” rather than grepping for the port. I’d expect it to give no big difference relative to what you got. > lsof -i Here again I expected you to look at the docs and infer that I meant for you to filter out only the interesting ports, 8080 in your case. -i:8080. But since we have all the TCP/IP connections, let’s see if we can learn something interesting from it. > fossil2 10468 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) > fossil2 10469 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) > fossil2 10470 myuser3u IPv4 35083942 0t0 TCP *:http-alt (LISTEN) etc. This is odd. It looks like Fossil is forking off children which listen and then never get anything. Yet, we see the same PID having many connections each already established. What’s the TCP connection rate to this machine? There must be nice tools for that which a network security admin would know about, but my off-the-cuff programmer brain brings up only this: $ sudo tshark -b duration:1 port 8080 and tcp.flags.syn==1 | wc -l Ignore the complaint about “multiple capture files”. We’re just wanting to know how many SYNs per second appear. Consider increasing it to 10 seconds or so to get a better baseline if the connection rate is in the single digits. > It seems some connections are just never closed for some reason. Right, which makes me wonder if you’ve got some botnet on your hands, banging on the server for no good reason. That’s why I asked for the connection rate. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
On 12/22/17, Olivier R.wrote: > > There is now more than 24 subprocesses of Fossil running, and it’s > getting really slow. Are you saying that these 24 Fossils are spinning - using CPU cycles - not just hung waiting on I/O? -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly
Le 19/12/2017 à 21:49, Warren Young a écrit : If it’s a sign of a bug, then it means something very bad has happened, like the network stack has lost track of its client somehow. To see that, you’d need to do a network capture on that fossil instance’s network sockets. Use netstat -nap or lsof -i to find out which TCP ports those are. There is now more than 24 subprocesses of Fossil running, and it’s getting really slow. netstat -na | grep :8080 tcp0 0 0.0.0.0:80800.0.0.0:* LISTEN tcp0 0 10.2.148.67:8080125.46.156.43:59187 ESTABLISHED tcp0 0 10.2.148.67:8080175.42.2.225:51516 ESTABLISHED tcp0 0 10.2.148.67:808041.243.10.196:17284 ESTABLISHED tcp0 0 10.2.148.67:8080182.138.162.53:64088 ESTABLISHED tcp0 0 10.2.148.67:8080125.119.221.109:33681 ESTABLISHED tcp0 0 10.2.148.67:8080221.13.12.222:34854 ESTABLISHED tcp0 0 10.2.148.67:808093.121.232.186:57361 ESTABLISHED tcp0 0 10.2.148.67:808092.145.13.163:38445 ESTABLISHED tcp0 0 10.2.148.67:808092.145.13.163:38444 ESTABLISHED tcp0 0 10.2.148.67:8080109.129.203.110:63628 ESTABLISHED tcp0 0 10.2.148.67:808045.79.223.204:41726 ESTABLISHED tcp0 0 10.2.148.67:8080221.13.12.26:38741 ESTABLISHED tcp0 0 10.2.148.67:8080112.80.138.198:54839 ESTABLISHED tcp0 0 10.2.148.67:8080106.114.63.0:17736 ESTABLISHED tcp0 0 10.2.148.67:808093.121.232.186:58675 ESTABLISHED tcp0 0 10.2.148.67:8080182.138.158.40:36008 ESTABLISHED tcp0 0 10.2.148.67:8080112.64.209.135:44966 ESTABLISHED tcp0 0 10.2.148.67:8080114.221.126.160:64177 ESTABLISHED tcp0 0 10.2.148.67:808037.170.252.107:19513 ESTABLISHED tcp0 0 10.2.148.67:8080124.235.138.25:64841 ESTABLISHED tcp0 0 10.2.148.67:8080221.13.12.209:50846 ESTABLISHED tcp0 0 10.2.148.67:8080139.227.182.157:55348 ESTABLISHED tcp0 0 10.2.148.67:808037.170.252.107:19514 ESTABLISHED tcp0 0 10.2.148.67:808066.249.64.74:61840 TIME_WAIT tcp0 0 10.2.148.67:8080109.129.203.110:63629 ESTABLISHED After clicking on “timeline” on the web ui. netstat -nap | grep :8080 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp0 0 0.0.0.0:80800.0.0.0:* LISTEN 10468/fossil2 tcp0 0 10.2.148.67:8080125.46.156.43:59187 ESTABLISHED 10468/fossil2 tcp0 0 10.2.148.67:8080175.42.2.225:51516 ESTABLISHED 10474/fossil2 tcp0 0 10.2.148.67:808041.243.10.196:17284 ESTABLISHED 20253/fossil2 tcp0 0 10.2.148.67:8080182.138.162.53:64088 ESTABLISHED 12334/fossil2 tcp0 0 10.2.148.67:8080125.119.221.109:33681 ESTABLISHED 12340/fossil2 tcp0 0 10.2.148.67:8080221.13.12.222:34854 ESTABLISHED 10477/fossil2 tcp0 0 10.2.148.67:808093.121.232.186:57361 ESTABLISHED 25582/fossil2 tcp0 0 10.2.148.67:808092.145.13.163:38445 ESTABLISHED 28559/fossil2 tcp0 0 10.2.148.67:808046.246.61.162:5743 TIME_WAIT - tcp0 0 10.2.148.67:808092.145.13.163:38444 ESTABLISHED 28558/fossil2 tcp0 0 10.2.148.67:8080109.129.203.110:63628 ESTABLISHED 15143/fossil2 tcp0 0 10.2.148.67:808045.79.223.204:41726 ESTABLISHED 31100/fossil2 tcp 389 0 10.2.148.67:808046.246.61.162:5751 CLOSE_WAIT - tcp0 0 10.2.148.67:8080221.13.12.26:38741 ESTABLISHED 12341/fossil2 tcp0 0 10.2.148.67:8080112.80.138.198:54839 ESTABLISHED 10469/fossil2 tcp 388 0 10.2.148.67:808046.246.61.162:5754 ESTABLISHED - tcp0 0 10.2.148.67:8080106.114.63.0:17736 ESTABLISHED 12335/fossil2 tcp0 0 10.2.148.67:808093.121.232.186:58675 ESTABLISHED 25581/fossil2 tcp0 0 10.2.148.67:8080182.138.158.40:36008 ESTABLISHED 10470/fossil2 tcp0 0 10.2.148.67:8080112.64.209.135:44966 ESTABLISHED 10471/fossil2 tcp0 0 10.2.148.67:8080114.221.126.160:64177 ESTABLISHED 12337/fossil2 tcp0 0 10.2.148.67:808037.170.252.107:19513 ESTABLISHED 30076/fossil2 tcp0 0 10.2.148.67:8080124.235.138.25:64841 ESTABLISHED 10475/fossil2 tcp0 0 10.2.148.67:8080221.13.12.209:50846 ESTABLISHED 12333/fossil2 tcp0 0 10.2.148.67:8080139.227.182.157:55348 ESTABLISHED
Re: [fossil-users] Digital signatures on check-ins. Was: tangent vs. wyoung on recent commti
On Dec 21, 2017, at 4:57 PM, jungle Boogiewrote: > > On 21 December 2017 at 15:03, Warren Young wrote: >> On Dec 21, 2017, at 2:58 PM, jungle Boogie wrote: >>> perhaps left of the username. >> >> I think right, simply because it’s currently ordered most-clicked to >> least-clicked, more or less. >> > > In between the username and date/time or right of date/time? I was thinking far right, after the tag in the Modern view. I sometimes click on the checkin ID, I almost never click on the user name, and when I click on a tag, it’s generally from some other view than the Timeline. I’m predicting that these icons will be even less-often clicked. You’ll want to see them far more often than you’ll want to poke into the details behind them. A detail view could occasionally be useful for roughly the same sorts of reasons that you occasionally want to look at some site’s TLS cert in a browser. New thought: Don’t bother making it configurable to hide these icons. That can be done at the skin level: svg.signState { display: none; } That doesn’t just make the icon invisible, it takes no space, so you don’t leave an icon-sized gap in the page layout. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Digital signatures on check-ins. Was: tangent vs. wyoung on recent commti
On Dec 21, 2017, at 4:03 PM, Warren Youngwrote: > > I’d recommend designing three distinct black-filled SVG icons Another idea: Unicode may have characters you can use as icons. Lock: http://www.fileformat.info/info/unicode/char/1f512/ Open: http://www.fileformat.info/info/unicode/char/1f513/ Roger: http://www.fileformat.info/info/unicode/char/2620/ That then requires that you’ve got local fonts that include these characters, of course. SVG could at least be embedded into the HTML, or into CSS via a content:url(“data:…”) reference: https://css-tricks.com/css-content/ ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users