Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Remco Schoen]

Hi,

I can confirm that this speeds up things.

Just noticed this thread and wondered if this could be as well the reason for 
my “slow" serving of unversioned files. When using docker images to host 
fossil, it uses fossil serve.

I just compiled the latest trunk on my Raspberry Pi and created a new docker 
image for it. Opening the website I have under the unversioned files went from 
2.6 s to 0.95 s! So there is definitely improvement here.

Kind regards,

Remco Schoen
 

> Op 23 dec. 2017, om 01:59 heeft Richard Hipp  het volgende 
> geschreven:
> 
> The latest trunk check-in on Fossil
> (http://www4.fossil-scm.org/info/05ec15cad53e8176) should do a much
> better job of handling load from "fossil server".  Please compile and
> run it and let me know what happens.
> 
> The www4.fossil-scm.org server is running as follows:
> 
>  /usr/local/bin/fossil serve -port 80 -nojail -max-latency 30
> /root/Fossils/fossil.fossil
> 
> That server is one of the €2.99/month servers available from
> www.scaleway.com - an ARM64 with 2GB of RAM located in the Paris
> datacenter.  I'll leave it running for a few days, but I do not intend
> for it to be a permanent mirror, so the link to www4.fossil-scm.org
> above may be dead if you are delayed in reading this posting.
> 
> -- 
> D. Richard Hipp
> d...@sqlite.org
> ___
> fossil-users mailing list
> fossil-users@lists.fossil-scm.org
> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Richard Hipp]

The latest trunk check-in on Fossil
(http://www4.fossil-scm.org/info/05ec15cad53e8176) should do a much
better job of handling load from "fossil server".  Please compile and
run it and let me know what happens.

The www4.fossil-scm.org server is running as follows:

  /usr/local/bin/fossil serve -port 80 -nojail -max-latency 30
/root/Fossils/fossil.fossil

That server is one of the €2.99/month servers available from
www.scaleway.com - an ARM64 with 2GB of RAM located in the Paris
datacenter.  I'll leave it running for a few days, but I do not intend
for it to be a permanent mirror, so the link to www4.fossil-scm.org
above may be dead if you are delayed in reading this posting.

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[John P. Rouillard]

In message ,
Warren Young writes:
>On Dec 22, 2017, at 10:20 AM, Olivier R.  wrote:
>> 
>> Le 22/12/2017 à 16:10, Warren Young a écrit :
>>> 1. Your repo is public-facing.  Is this a reasonable number of
>>> clients to be connected at any given time to this repo?  It seems
>>> high to me, given the transient nature of most Fossil connections.
>> 
>> Only two devs have the right to modify the online repository.
>> I’d be surprised if there were more than 5 people connected to this
>> repo at the same time.
>
>Your first netstat -na output shows 24 established connections, most
>to distinct remote peer IPs.
>
>Additionally, Fossil connections tend to be short-lived due to the
>nature of Fossil.  [...]
>Therefore, one of two things is happening:
>
>1. You have a high rate of connections to the server, so that at any
>   one time, we can see dozens of them.  Disproving this hypothesis is
>   the purpose of the tshark test.
>
>2. You have connections that stay open for a long time, which suggests:
>
>2a. A connection handle leak.  But if this were common, lots of
>people would have found it before you.
>
>2b. Bad actors on those remote hosts, which is why I bought up the
>possibility of a botnet attack.
>
>If we have a 2b case, then you’d want to find some way to identify
>the connections somehow, so that you can block them.

The only way I managed to reproduce the lsof output that was reported
earlier was to connect to my fossil using netcat and send no
data. Once I sent any data (by hitting a return) fossil reported a 403
(since I sent no http headers).

As I noted earlier I have hiawatha running in front of the fossil
server.  It kills idle connections after I think 60 seconds. So if
there is no traffic, the connection to the back end fossil server will
be killed.

Note that Warren is running nginx as the front end for his fossil
server. I think that has anti-DOS and timeouts on client header/body
and response times that can do something similar.

Does fossil have any idle connection timeout limits?

--
-- rouilj
John Rouillard
===
My employers don't acknowledge my existence much less my opinions.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Olivier R.]

Le 22/12/2017 à 21:34, Richard Hipp a écrit :

On 12/22/17, Olivier R.  wrote:


I also run Fossil on a cheap VPS.
https://www.scaleway.com/virtual-cloud-servers/
(The starter version at €2.99/month)



I'm building up a new Fossil server on this VPS now - on an ARM64-2G
machine in Paris.  It is slow-going.  Apparently the 2.99/mo ARM64
machines are not very fast :-)


I’m not using the ARM version but the x86-64 version.
And it’s fast enough for me and fits my needs. My repo is small: only 
130 Gb.


Note that the bug usually happens after weeks or months, but then it 
slows down quite quickly as new processes seem more likely to pop up 
when it has begun.


Olivier
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Olivier R.]

Le 22/12/2017 à 22:39, Richard Hipp a écrit :

On 12/22/17, Olivier R.  wrote:


I also run Fossil on a cheap VPS.


Maybe you have used up your bandwidth quota and the ISP is throttling
your connection?



There is no bandwidth quota.
And when I kill the processes and relaunch Fossil, it runs quickly.


Olivier
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Richard Hipp]

On 12/22/17, Olivier R.  wrote:
>
> I also run Fossil on a cheap VPS.

Maybe you have used up your bandwidth quota and the ISP is throttling
your connection?

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Richard Hipp]

On 12/22/17, Richard Hipp  wrote:
> On 12/22/17, Olivier R.  wrote:
>>
>> I also run Fossil on a cheap VPS.
>> https://www.scaleway.com/virtual-cloud-servers/
>> (The starter version at €2.99/month)
>>
>
> I'm building up a new Fossil server on this VPS now

The new server is now up and running.  Please visit http://www4.fossil-scm.org/

I compiled the latest trunk version using "./configure
--with-openssl=none --no-opt".  I'll try to keep an eye on this system
to see how much CPU and memory it is using.

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Richard Hipp]

On 12/22/17, Olivier R.  wrote:
>
> I also run Fossil on a cheap VPS.
> https://www.scaleway.com/virtual-cloud-servers/
> (The starter version at €2.99/month)
>

I'm building up a new Fossil server on this VPS now - on an ARM64-2G
machine in Paris.  It is slow-going.  Apparently the 2.99/mo ARM64
machines are not very fast :-)

Assuming I get this running, I'll start up a "fossil server" and let
people pound on it, in order to try to repro the problem.

I'll let you know once the build finishes

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Olivier R.]

Le 22/12/2017 à 18:41, Warren Young a écrit :


I’ve added the user to the “wireshark” group, but it doesn’t work.


You have to log out and back in before group changes will take
effect.


That’s what I did. But it didn’t work.

So I used `lsof -i:8080` again.

Here is the difference with the previous connections:
https://www.diffchecker.com/KRGBfoK1

So it seems that almost all connections seen few hours ago are still there.


Olivier
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Olivier R.]

Le 22/12/2017 à 19:57, Warren Young a écrit :

On Dec 22, 2017, at 9:03 AM, Warren Young 
wrote:


Olivier, what about memory usage for the Fossil processes?  “top”
can give you this.


I just checked my public server, and each of the 4 Fossil instances
is taking about 20 MB of VM, of which only about 800 kB is resident.
top calculates this as 0.2% of the server’s small allocation of
virtual RAM.  (It’s a cheap VPS, not a real server.)


I also run Fossil on a cheap VPS.
https://www.scaleway.com/virtual-cloud-servers/
(The starter version at €2.99/month)

There is no memory overload (it seems).

top - 19:40:57 up 238 days,  9:50,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 110 total,   1 running, 108 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0.2 us,  0.0 sy,  0.0 ni, 99.3 id,  0.0 wa,  0.0 hi,  0.0 si, 
0.5 st

KiB Mem:   2049824 total,  1782024 used,   267800 free,   155636 buffers
KiB Swap:0 total,0 used,0 free.  1436172 cached Mem

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
10468 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
10469 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
10470 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
10471 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
10474 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
10475 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
10477 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
12333 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
12334 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
12335 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
12336 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
12337 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
12340 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
12341 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 fossil2
15143 myuser20   0   14876   3452   2724 S   0.0  0.2   0:00.00 fossil2
15144 myuser20   0   14876   3452   2724 S   0.0  0.2   0:00.00 fossil2
17096 myuser20   0   80656   4420   3568 S   0.0  0.2   0:00.01 sshd
17097 myuser20   0   12708   1760   1600 S   0.0  0.1   0:00.00 
sftp-server

17098 myuser20   0   22784   4640   3340 S   0.0  0.2   0:00.09 bash
17107 myuser20   0   0  0  0 Z   0.0  0.0   0:00.01 fossil2
17108 myuser20   0   23628   2896   2440 R   0.0  0.1   0:00.04 top
17876 myuser20   0   14872   4248   3524 S   0.0  0.2   0:41.50 fossil2
20253 myuser20   0   14876   3452   2724 S   0.0  0.2   0:00.00 fossil2
25581 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 fossil2
25582 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 fossil2
28558 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 fossil2
28559 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 fossil2
30076 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 fossil2
30077 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 fossil2
31100 myuser20   0   14876   3452   2724 S   0.0  0.2   0:00.00 fossil2


Olivier
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 22, 2017, at 9:03 AM, Warren Young  wrote:
> 
> Olivier, what about memory usage for the Fossil processes?  “top” can give 
> you this.

I just checked my public server, and each of the 4 Fossil instances is taking 
about 20 MB of VM, of which only about 800 kB is resident.  top calculates this 
as 0.2% of the server’s small allocation of virtual RAM.  (It’s a cheap VPS, 
not a real server.)

I’ve taken the opportunity to upgrade, which changes VM not a whit; RSS went up 
to ~1.7 MB, doubtless to be whittled down over time by the VMM; and /timeline 
takes 8 ms now.

So no, my server hadn’t been slowed down at all in that 11 week 4 day span.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 22, 2017, at 11:31 AM, Warren Young  wrote:
> 
> I posted a long HOWTO about it back in June of 2016:
> 
>   https://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg23320.html

I just realized that that message is a reply to my post, not the post itself.  
Here’s the direct link:

https://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg22907.html

It’s a fair bit of work, but it’s worth it for TLS protection of login 
credentials alone.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 22, 2017, at 10:53 AM, Tomek Kott  wrote:
> 
> FWIW I also run in server mode constantly because its easy. My users (there 
> are 10, they probably hit the server in bursts of 10 times one one day of the 
> week) occasionally notice slowdowns and ping me.

When I said above that I occasionally restart mine, I meant only for upgrades 
to newer versions of Fossils.  I can’t ever remember restarting it because it 
was slow.

My public Fossil server was last restarted on October 2.  I just pulled up a 
timeline in 7ms.

My public server has 4 Fossil instances, one for each repo it serves.  (Not 
important why.)  Each instance is bound to localhost, with an nginx proxy in 
front of them.  I’ve just checked, and only those 4 processes are currently 
running, no children.

They’re running in SCGI mode, for whatever that’s worth.

Olivier, if you want to try my configuration, I posted a long HOWTO about it 
back in June of 2016:

   https://www.mail-archive.com/fossil-users@lists.fossil-scm.org/msg23320.html

It’s obsolete in places because nginx + Let’s Encrypt certificates no longer 
require manual updates.  I won’t be motivated to rewrite it until I move to 
another hosting provider, which will force me to rebuild my site configuration.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Tomek Kott]

FWIW I also run in server mode constantly because its easy. My users (there are 
10, they probably hit the server in bursts of 10 times one one day of the week) 
occasionally notice slowdowns and ping me. This is for an internal site only, 
no public facing, so shouldn't be huge numbers of users.


I restart the server and all is well. I don't see this as a problem since I 
know I should be running a 'real' server, but *shrug* this is good enough and a 
simple restart works well.


Tomek



From: fossil-users  on behalf of 
fossil-users-requ...@lists.fossil-scm.org 

Sent: Friday, December 22, 2017 12:41 PM
To: fossil-users@lists.fossil-scm.org
Subject: fossil-users Digest, Vol 119, Issue 49

In message 

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 22, 2017, at 10:20 AM, Olivier R.  wrote:
> 
> Le 22/12/2017 à 16:10, Warren Young a écrit :
> 
>> 1. Your repo is public-facing.  Is this a reasonable number of
>> clients to be connected at any given time to this repo?  It seems
>> high to me, given the transient nature of most Fossil connections.
> 
> Only two devs have the right to modify the online repository.
> I’d be surprised if there were more than 5 people connected to this repo at 
> the same time.

Your first netstat -na output shows 24 established connections, most to 
distinct remote peer IPs.

Additionally, Fossil connections tend to be short-lived due to the nature of 
Fossil.  Even if you have someone “actively” working with Fossil, the 
connections will come and go rapidly in a series, not have a single long-lived 
connection.

Therefore, one of two things is happening:

1. You have a high rate of connections to the server, so that at any one time, 
we can see dozens of them.  Disproving this hypothesis is the purpose of the 
tshark test.

2. You have connections that stay open for a long time, which suggests:

2a. A connection handle leak.  But if this were common, lots of people would 
have found it before you.

2b. Bad actors on those remote hosts, which is why I bought up the possibility 
of a botnet attack.

If we have a 2b case, then you’d want to find some way to identify the 
connections somehow, so that you can block them.

> I assume that people who go to dev section are not the majority. Maybe 20 
> visitors per day if I’m optimistic…

20 *legitimate* visitors.  That doesn’t count those hitting your site for no 
good reason, which does happen on the wild Internet.

> I’ve added the user to the “wireshark” group, but it doesn’t work.

You have to log out and back in before group changes will take effect.

> Same error message if i run this as root.

Better to follow tshark’s advice and run it as a normal user, with group 
privileges allowing network capture.

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Olivier R.]

Le 22/12/2017 à 16:10, Warren Young a écrit :


1. Your repo is public-facing.  Is this a reasonable number of
clients to be connected at any given time to this repo?  It seems
high to me, given the transient nature of most Fossil connections.


Only two devs have the right to modify the online repository.
I’d be surprised if there were more than 5 people connected to this repo 
at the same time.




lsof -i


Here again I expected you to look at the docs and infer that I meant
for you to filter out only the interesting ports, 8080 in your case.
-i:8080.


Hmm. Sorry.
As I said before, this server only runs Fossil and has no other purpose.

lsof -i:8080

COMMAND   PID   USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
fossil2 10468 myuser0u  IPv4 42925675  0t0  TCP 
scw-f45a9f:http-alt->hn.kd.ny.adsl:59187 (ESTABLISHED)
fossil2 10468 myuser1u  IPv4 42925675  0t0  TCP 
scw-f45a9f:http-alt->hn.kd.ny.adsl:59187 (ESTABLISHED)
fossil2 10468 myuser2u  IPv4 42925675  0t0  TCP 
scw-f45a9f:http-alt->hn.kd.ny.adsl:59187 (ESTABLISHED)

fossil2 10468 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
fossil2 10469 myuser0u  IPv4 42925676  0t0  TCP 
scw-f45a9f:http-alt->112.80.138.198:54839 (ESTABLISHED)
fossil2 10469 myuser1u  IPv4 42925676  0t0  TCP 
scw-f45a9f:http-alt->112.80.138.198:54839 (ESTABLISHED)
fossil2 10469 myuser2u  IPv4 42925676  0t0  TCP 
scw-f45a9f:http-alt->112.80.138.198:54839 (ESTABLISHED)

fossil2 10469 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
fossil2 10470 myuser0u  IPv4 42925678  0t0  TCP 
scw-f45a9f:http-alt->182.138.158.40:36008 (ESTABLISHED)
fossil2 10470 myuser1u  IPv4 42925678  0t0  TCP 
scw-f45a9f:http-alt->182.138.158.40:36008 (ESTABLISHED)
fossil2 10470 myuser2u  IPv4 42925678  0t0  TCP 
scw-f45a9f:http-alt->182.138.158.40:36008 (ESTABLISHED)

fossil2 10470 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
fossil2 10471 myuser0u  IPv4 42925679  0t0  TCP 
scw-f45a9f:http-alt->112.64.209.135:44966 (ESTABLISHED)
fossil2 10471 myuser1u  IPv4 42925679  0t0  TCP 
scw-f45a9f:http-alt->112.64.209.135:44966 (ESTABLISHED)
fossil2 10471 myuser2u  IPv4 42925679  0t0  TCP 
scw-f45a9f:http-alt->112.64.209.135:44966 (ESTABLISHED)

fossil2 10471 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
fossil2 10474 myuser0u  IPv4 42925688  0t0  TCP 
scw-f45a9f:http-alt->175.42.2.225:51516 (ESTABLISHED)
fossil2 10474 myuser1u  IPv4 42925688  0t0  TCP 
scw-f45a9f:http-alt->175.42.2.225:51516 (ESTABLISHED)
fossil2 10474 myuser2u  IPv4 42925688  0t0  TCP 
scw-f45a9f:http-alt->175.42.2.225:51516 (ESTABLISHED)

fossil2 10474 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
fossil2 10475 myuser0u  IPv4 42925700  0t0  TCP 
scw-f45a9f:http-alt->124.235.138.25:64841 (ESTABLISHED)
fossil2 10475 myuser1u  IPv4 42925700  0t0  TCP 
scw-f45a9f:http-alt->124.235.138.25:64841 (ESTABLISHED)
fossil2 10475 myuser2u  IPv4 42925700  0t0  TCP 
scw-f45a9f:http-alt->124.235.138.25:64841 (ESTABLISHED)

fossil2 10475 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
fossil2 10477 myuser0u  IPv4 42925703  0t0  TCP 
scw-f45a9f:http-alt->221.13.12.222:34854 (ESTABLISHED)
fossil2 10477 myuser1u  IPv4 42925703  0t0  TCP 
scw-f45a9f:http-alt->221.13.12.222:34854 (ESTABLISHED)
fossil2 10477 myuser2u  IPv4 42925703  0t0  TCP 
scw-f45a9f:http-alt->221.13.12.222:34854 (ESTABLISHED)

fossil2 10477 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
fossil2 12333 myuser0u  IPv4 42952323  0t0  TCP 
scw-f45a9f:http-alt->221.13.12.209:50846 (ESTABLISHED)
fossil2 12333 myuser1u  IPv4 42952323  0t0  TCP 
scw-f45a9f:http-alt->221.13.12.209:50846 (ESTABLISHED)
fossil2 12333 myuser2u  IPv4 42952323  0t0  TCP 
scw-f45a9f:http-alt->221.13.12.209:50846 (ESTABLISHED)

fossil2 12333 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
fossil2 12334 myuser0u  IPv4 42952324  0t0  TCP 
scw-f45a9f:http-alt->182.138.162.53:64088 (ESTABLISHED)
fossil2 12334 myuser1u  IPv4 42952324  0t0  TCP 
scw-f45a9f:http-alt->182.138.162.53:64088 (ESTABLISHED)
fossil2 12334 myuser2u  IPv4 42952324  0t0  TCP 
scw-f45a9f:http-alt->182.138.162.53:64088 (ESTABLISHED)

fossil2 12334 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
fossil2 12335 myuser0u  IPv4 42952325  0t0  TCP 
scw-f45a9f:http-alt->106.114.63.0:17736 (ESTABLISHED)
fossil2 12335 myuser1u  IPv4 42952325  0t0  TCP 
scw-f45a9f:http-alt->106.114.63.0:17736 (ESTABLISHED)
fossil2 12335 myuser2u  IPv4 42952325  0t0  TCP 
scw-f45a9f:http-alt->106.114.63.0:17736 (ESTABLISHED)

fossil2 12335 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
fossil2 12336 myuser0u  IPv4 42952329  0t0  TCP 
scw-f45a9f:http-alt->139.227.182.157:55348 (ESTABLISHED)
fossil2 12336 myuser 

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[John P. Rouillard]

Hello all:

In message 

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 22, 2017, at 8:56 AM, Richard Hipp  wrote:
> 
> (afaik) nobody uses "fossil server"
> for long-running websites.

I do.  Mine stay running for months at a time, including public-facing ones.

> use the xinetd/inetd style of invoking Fossil

I can only see that helping if there is some kind of leak.  We can see from his 
test results that there is no socket descriptor leak.

Olivier, what about memory usage for the Fossil processes?  “top” can give you 
this.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Richard Hipp]

On 12/22/17, Olivier R.  wrote:
>
> I didn’t use Fossil 2.3 because the SQLite version bundled was in beta
> stage.

That is NOT a good reason to reject the use of Fossil 2.3.
-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Richard Hipp]

On 12/22/17, Olivier R.  wrote:
>
> I’ll have to kill all these processes soon and relaunch Fossil.
> I’ll do it when you think you have enough information about this
> problem. I’ll use Fossil 2.4 next time.
>

Maybe there is some kind of issue in the HTTP server implementation in
cgi.c that causes it to slow down after running for a long time.  This
might have gone unnoticed, since (afaik) nobody uses "fossil server"
for long-running websites.

I will investigate and try to repro the problem when I have time.

In the meantime, please consider setting up your website to use the
xinetd/inetd style of invoking Fossil, as described in the
https://www.fossil-scm.org/fossil/doc/trunk/www/server.wiki document.
This is what Fossil itself uses, and also SQLite. It has been running
for years under heavy load without issues.  We know that method works
well.

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 22, 2017, at 8:24 AM, Olivier R.  wrote:
> 
> I’ll have to kill all these processes soon and relaunch Fossil.
> I’ll do it when you think you have enough information about this problem.

My request for connection rates doesn’t require that you keep Fossil running — 
in fact it would give correct results even if Fossil is down! — but drh may 
require state you can only learn from the executable still running.

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Olivier R.]

Le 22/12/2017 à 15:59, Richard Hipp a écrit :

On 12/22/17, Olivier R.  wrote:


There is now more than 24 subprocesses of Fossil running, and it’s
getting really slow.


Are you saying that these 24 Fossils are spinning - using CPU cycles -
not just hung waiting on I/O?



These processes doesn’t seem to use much CPU time, but actually 
interaction with Fossil is now *very* slow.


top - 15:19:22 up 238 days,  5:28,  1 user,  load average: 0.00, 0.00, 0.00
Tasks: 111 total,   1 running, 109 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0.0 us,  0.2 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si, 
0.3 st

KiB Mem:   2049824 total,  1650376 used,   399448 free,   154816 buffers
KiB Swap:0 total,0 used,0 free.  1308056 cached Mem

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
12873 myuser20   0   80660   4540   3696 S   0.0  0.2   0:00.11 
`- sshd
13229 myuser20   0   80660   4456   3608 S   0.0  0.2   0:00.02 
`- sshd
13230 myuser20   0   12708   1780   1624 S   0.0  0.1   0:00.00 
`- sftp-server
13231 myuser20   0   22784   4564   3268 S   0.0  0.2   0:00.09 
`- bash
13241 myuser20   0   23628   3012   2456 R   0.3  0.1   0:00.38 
`- top
17876 myuser20   0   14872   4248   3524 S   0.0  0.2   0:41.40  `- 
fossil2
31100 myuser20   0   14876   3452   2724 S   0.0  0.2   0:00.00 
`- fossil2
30076 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 
`- fossil2
30077 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 
`- fossil2
20253 myuser20   0   14876   3452   2724 S   0.0  0.2   0:00.00 
`- fossil2
25581 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 
`- fossil2
25582 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 
`- fossil2
15143 myuser20   0   14876   3452   2724 S   0.0  0.2   0:00.00 
`- fossil2
15144 myuser20   0   14876   3452   2724 S   0.0  0.2   0:00.00 
`- fossil2
28558 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 
`- fossil2
28559 myuser20   0   14876   3448   2720 S   0.0  0.2   0:00.00 
`- fossil2
10468 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
10469 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
10470 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
10471 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
10474 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
10475 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
10477 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
12333 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
12334 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
12335 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
12336 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
12337 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
12340 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
12341 myuser20   0   14876   3516   2784 S   0.0  0.2   0:00.00 
`- fossil2
13247 myuser20   0   0  0  0 Z   0.0  0.0   0:00.01 
`- fossil2


You can see how slow it is by going there:
http://212.47.254.152:8080

In normal times, it works fast.

I’ll have to kill all these processes soon and relaunch Fossil.
I’ll do it when you think you have enough information about this 
problem. I’ll use Fossil 2.4 next time.


Regards,
Olivier
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 22, 2017, at 8:10 AM, Warren Young  wrote:
> 
>   $ sudo tshark -b duration:1 port 8080 and tcp.flags.syn==1 | wc -l

Sorry, that’s bogus.  Try this instead:

$ sudo tshark -i enp3s0 -w x.pcap -b duration:5 -a files:1 \
  port 8080 and "tcp[tcpflags] & tcp-syn != 0”

Adjust the interface name (-i) to suit your machine.

The capture file x.pcap is just there to placate tshark.  You can remove it 
when the command finishes, because what we’re interested in is the number that 
tshark reports.  That’s also why we don’t need wc -l.

And as for the SYN filter, that was a display filter, not a capture filter.  
Sigh.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 22, 2017, at 7:46 AM, Olivier R.  wrote:
> 
> Le 19/12/2017 à 21:49, Warren Young a écrit :
> 
>> If it’s a sign of a bug, then it means something very bad has
>> happened, like the network stack has lost track of its client
>> somehow.  To see that, you’d need to do a network capture on that
>> fossil instance’s network sockets.  Use netstat -nap or lsof -i to
>> find out which TCP ports those are.
> 
> There is now more than 24 subprocesses of Fossil running, and it’s getting 
> really slow.
> 
> netstat -na | grep :8080
> 
> tcp0  0 0.0.0.0:80800.0.0.0:*   LISTEN
> tcp0  0 10.2.148.67:8080125.46.156.43:59187 ESTABLISHED
> tcp0  0 10.2.148.67:8080175.42.2.225:51516 ESTABLISHED

Two things I learn from this:

1. Your repo is public-facing.  Is this a reasonable number of clients to be 
connected at any given time to this repo?  It seems high to me, given the 
transient nature of most Fossil connections.

2. I don’t see many WAIT states, so we’re probably not looking at the sort of 
weird bug as in that Cloudflare blog post I linked to earlier.

> After clicking on “timeline” on the web ui.
> 
> netstat -nap | grep :8080

The -p in that command was intended to let you find the Fossil process by name, 
rather than by port.  So, “netstat -nap | grep fossil” rather than grepping for 
the port.

I’d expect it to give no big difference relative to what you got.

> lsof -i

Here again I expected you to look at the docs and infer that I meant for you to 
filter out only the interesting ports, 8080 in your case.  -i:8080.

But since we have all the TCP/IP connections, let’s see if we can learn 
something interesting from it.

> fossil2 10468 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
> fossil2 10469 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)
> fossil2 10470 myuser3u  IPv4 35083942  0t0  TCP *:http-alt (LISTEN)

etc.  This is odd.  It looks like Fossil is forking off children which listen 
and then never get anything.  Yet, we see the same PID having many connections 
each already established.

What’s the TCP connection rate to this machine?

There must be nice tools for that which a network security admin would know 
about, but my off-the-cuff programmer brain brings up only this:

   $ sudo tshark -b duration:1 port 8080 and tcp.flags.syn==1 | wc -l

Ignore the complaint about “multiple capture files”.  We’re just wanting to 
know how many SYNs per second appear.  Consider increasing it to 10 seconds or 
so to get a better baseline if the connection rate is in the single digits.

> It seems some connections are just never closed for some reason.

Right, which makes me wonder if you’ve got some botnet on your hands, banging 
on the server for no good reason.

That’s why I asked for the connection rate.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Richard Hipp]

On 12/22/17, Olivier R.  wrote:
>
> There is now more than 24 subprocesses of Fossil running, and it’s
> getting really slow.

Are you saying that these 24 Fossils are spinning - using CPU cycles -
not just hung waiting on I/O?
-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Olivier R.]

Le 19/12/2017 à 21:49, Warren Young a écrit :



If it’s a sign of a bug, then it means something very bad has
happened, like the network stack has lost track of its client
somehow.  To see that, you’d need to do a network capture on that
fossil instance’s network sockets.  Use netstat -nap or lsof -i to
find out which TCP ports those are.


There is now more than 24 subprocesses of Fossil running, and it’s 
getting really slow.


netstat -na | grep :8080

tcp0  0 0.0.0.0:80800.0.0.0:*   LISTEN
tcp0  0 10.2.148.67:8080125.46.156.43:59187 
ESTABLISHED
tcp0  0 10.2.148.67:8080175.42.2.225:51516 
ESTABLISHED
tcp0  0 10.2.148.67:808041.243.10.196:17284 
ESTABLISHED
tcp0  0 10.2.148.67:8080182.138.162.53:64088 
ESTABLISHED
tcp0  0 10.2.148.67:8080125.119.221.109:33681 
ESTABLISHED
tcp0  0 10.2.148.67:8080221.13.12.222:34854 
ESTABLISHED
tcp0  0 10.2.148.67:808093.121.232.186:57361 
ESTABLISHED
tcp0  0 10.2.148.67:808092.145.13.163:38445 
ESTABLISHED
tcp0  0 10.2.148.67:808092.145.13.163:38444 
ESTABLISHED
tcp0  0 10.2.148.67:8080109.129.203.110:63628 
ESTABLISHED
tcp0  0 10.2.148.67:808045.79.223.204:41726 
ESTABLISHED
tcp0  0 10.2.148.67:8080221.13.12.26:38741 
ESTABLISHED
tcp0  0 10.2.148.67:8080112.80.138.198:54839 
ESTABLISHED
tcp0  0 10.2.148.67:8080106.114.63.0:17736 
ESTABLISHED
tcp0  0 10.2.148.67:808093.121.232.186:58675 
ESTABLISHED
tcp0  0 10.2.148.67:8080182.138.158.40:36008 
ESTABLISHED
tcp0  0 10.2.148.67:8080112.64.209.135:44966 
ESTABLISHED
tcp0  0 10.2.148.67:8080114.221.126.160:64177 
ESTABLISHED
tcp0  0 10.2.148.67:808037.170.252.107:19513 
ESTABLISHED
tcp0  0 10.2.148.67:8080124.235.138.25:64841 
ESTABLISHED
tcp0  0 10.2.148.67:8080221.13.12.209:50846 
ESTABLISHED
tcp0  0 10.2.148.67:8080139.227.182.157:55348 
ESTABLISHED
tcp0  0 10.2.148.67:808037.170.252.107:19514 
ESTABLISHED
tcp0  0 10.2.148.67:808066.249.64.74:61840 
TIME_WAIT
tcp0  0 10.2.148.67:8080109.129.203.110:63629 
ESTABLISHED


After clicking on “timeline” on the web ui.

netstat -nap | grep :8080

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp0  0 0.0.0.0:80800.0.0.0:* 
LISTEN  10468/fossil2
tcp0  0 10.2.148.67:8080125.46.156.43:59187 
ESTABLISHED 10468/fossil2
tcp0  0 10.2.148.67:8080175.42.2.225:51516 
ESTABLISHED 10474/fossil2
tcp0  0 10.2.148.67:808041.243.10.196:17284 
ESTABLISHED 20253/fossil2
tcp0  0 10.2.148.67:8080182.138.162.53:64088 
ESTABLISHED 12334/fossil2
tcp0  0 10.2.148.67:8080125.119.221.109:33681 
ESTABLISHED 12340/fossil2
tcp0  0 10.2.148.67:8080221.13.12.222:34854 
ESTABLISHED 10477/fossil2
tcp0  0 10.2.148.67:808093.121.232.186:57361 
ESTABLISHED 25582/fossil2
tcp0  0 10.2.148.67:808092.145.13.163:38445 
ESTABLISHED 28559/fossil2
tcp0  0 10.2.148.67:808046.246.61.162:5743 
TIME_WAIT   -
tcp0  0 10.2.148.67:808092.145.13.163:38444 
ESTABLISHED 28558/fossil2
tcp0  0 10.2.148.67:8080109.129.203.110:63628 
ESTABLISHED 15143/fossil2
tcp0  0 10.2.148.67:808045.79.223.204:41726 
ESTABLISHED 31100/fossil2
tcp  389  0 10.2.148.67:808046.246.61.162:5751 
CLOSE_WAIT  -
tcp0  0 10.2.148.67:8080221.13.12.26:38741 
ESTABLISHED 12341/fossil2
tcp0  0 10.2.148.67:8080112.80.138.198:54839 
ESTABLISHED 10469/fossil2
tcp  388  0 10.2.148.67:808046.246.61.162:5754 
ESTABLISHED -
tcp0  0 10.2.148.67:8080106.114.63.0:17736 
ESTABLISHED 12335/fossil2
tcp0  0 10.2.148.67:808093.121.232.186:58675 
ESTABLISHED 25581/fossil2
tcp0  0 10.2.148.67:8080182.138.158.40:36008 
ESTABLISHED 10470/fossil2
tcp0  0 10.2.148.67:8080112.64.209.135:44966 
ESTABLISHED 10471/fossil2
tcp0  0 10.2.148.67:8080114.221.126.160:64177 
ESTABLISHED 12337/fossil2
tcp0  0 10.2.148.67:808037.170.252.107:19513 
ESTABLISHED 30076/fossil2
tcp0  0 10.2.148.67:8080124.235.138.25:64841 
ESTABLISHED 10475/fossil2
tcp0  0 10.2.148.67:8080221.13.12.209:50846 
ESTABLISHED 12333/fossil2
tcp0  0 10.2.148.67:8080139.227.182.157:55348 
ESTABLISHED 

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 19, 2017, at 1:49 PM, Warren Young  wrote:
> 
> If it’s a sign of a bug, then it means something very bad has happened, like 
> the network stack has lost track of its client somehow.  To see that, you’d 
> need to do a network capture on that fossil instance’s network sockets.  Use 
> netstat -nap or lsof -i to find out which TCP ports those are.

Something else which might be helpful is the output of:

$ netstat -na | grep :80

That’s assuming Fossil listens on TCP port 80 on your system.  If not, adjust 
to suit.

This possibly-relevant and certainly-fascinating article just popped up on 
Reddit recently:


https://blog.cloudflare.com/this-is-strictly-a-violation-of-the-tcp-specification/

TCP can do some really strange things in the corner cases.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 19, 2017, at 5:20 AM, Olivier R.  wrote:
> 
> #0  0x7fd8ff85e873 in __select_nocancel () at 
> ../sysdeps/unix/syscall-template.S:81
> #0  0x7fd8ff85e873 in __select_nocancel () at 
> ../sysdeps/unix/syscall-template.S:81
> #0  0x7fd8ff858ba0 in __read_nocancel () at 
> ../sysdeps/unix/syscall-template.S:81
> #0  0x7fd8ff858ba0 in __read_nocancel () at 
> ../sysdeps/unix/syscall-template.S:81
> #0  0x7fd8ff858ba0 in __read_nocancel () at 
> ../sysdeps/unix/syscall-template.S:81
> #0  0x7fd8ff858ba0 in __read_nocancel () at 
> ../sysdeps/unix/syscall-template.S:81

These all mean that the processes are blocked on network I/O, which is the 
typical state for a network server like Fossil.  It could be fine, and we’re 
chasing the wrong thing.

If it’s a sign of a bug, then it means something very bad has happened, like 
the network stack has lost track of its client somehow.  To see that, you’d 
need to do a network capture on that fossil instance’s network sockets.  Use 
netstat -nap or lsof -i to find out which TCP ports those are.

>> What does “fossil ver” say?  I ask because line numbers are more
>> helpful if you also give the checkin ID they refer to. 
> 
> This is fossil version 2.2 [81d7d3f43e] 2017-04-11 20:54:55 UTC

Any particular reason you’re not running 2.4?  I don’t recall any networking 
bugs being fixed in the past 2 releases, but who knows?

Besides, there’s some nice stuff in those two releases.

> Assuming that new PIDs are higher than the previous ones

They will be, until the PID counter wraps around.  I believe Linux normally 
uses a 16-bit counter.

macOS, oddly, uses a 32-bit counter but wraps at 9, so you still only get 5 
digits.

And then there are the OSes that randomize PIDs…

> it is interesting to notice that two of the subprocesses have a lower PID 
> than the main one.

Not necessarily.  Hit it with another connection.  If you get another in the 
15-16k range, then you know the PID counter has wrapped since PID 17876 was 
spawned.

You can also use a PID tree view, as top or htop can give, so you don’t ascribe 
too much meaning to PIDs.  PPIDs are more interesting, but that doesn’t show up 
in your output.  A tree view effectively encodes the same information.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Olivier R.]

Le 19/12/2017 à 02:00, Warren Young a écrit :

On Dec 18, 2017, at 6:52 AM, Olivier R.  wrote:


When gdb was active, Fossil didn’t answer when asking for a
webpage. It seemed blocked.


That’s exactly what happens.  If you want the process to run while
GDB remains attached, say “cont” after attaching.


Here is what gdb said:


Did it say the same thing if you reattached each time?


New try:

(gdb) bt
#0  0x7fd8ff85e873 in __select_nocancel () at 
../sysdeps/unix/syscall-template.S:81
#1  0x00416162 in cgi_http_server (mnPort=mnPort@entry=8080, 
mxPort=mxPort@entry=8180,
zBrowser=, zBrowser@entry=0x0, 
zIpAddr=zIpAddr@entry=0x0, flags=flags@entry=12)

at ./src/cgi.c:1845
#2  0x0044f92a in cmd_webserver () at ./src/main.c:2493
#3  0x00407fec in main (argc=, argv=out>) at ./src/main.c:760


New try (after performing some actions):

(gdb) bt
#0  0x7fd8ff85e873 in __select_nocancel () at 
../sysdeps/unix/syscall-template.S:81
#1  0x00416162 in cgi_http_server (mnPort=mnPort@entry=8080, 
mxPort=mxPort@entry=8180,
zBrowser=, zBrowser@entry=0x0, 
zIpAddr=zIpAddr@entry=0x0, flags=flags@entry=12)

at ./src/cgi.c:1845
#2  0x0044f92a in cmd_webserver () at ./src/main.c:2493
#3  0x00407fec in main (argc=, argv=out>) at ./src/main.c:760



I also launched gdb on different subprocesses:

(gdb) bt
#0  0x7fd8ff858ba0 in __read_nocancel () at 
../sysdeps/unix/syscall-template.S:81
#1  0x7fd8ff7f23a0 in _IO_new_file_underflow (fp=0x7fd8ffb234e0 
<_IO_2_1_stdin_>) at fileops.c:605
#2  0x7fd8ff7f31ce in __GI__IO_default_uflow (fp=0x7fd8ffb234e0 
<_IO_2_1_stdin_>) at genops.c:435
#3  0x7fd8ff7e7f84 in __GI__IO_getline_info 
(fp=fp@entry=0x7fd8ffb234e0 <_IO_2_1_stdin_>,
buf=buf@entry=0x7ffd56077bf0 "GET", n=n@entry=1999, 
delim=delim@entry=10, extract_delim=extract_delim@entry=1,

eof=eof@entry=0x0) at iogetline.c:69
#4  0x7fd8ff7e8078 in __GI__IO_getline (fp=fp@entry=0x7fd8ffb234e0 
<_IO_2_1_stdin_>,
buf=buf@entry=0x7ffd56077bf0 "GET", n=n@entry=1999, 
delim=delim@entry=10, extract_delim=extract_delim@entry=1)

at iogetline.c:38
#5  0x7fd8ff7e6e96 in _IO_fgets (buf=buf@entry=0x7ffd56077bf0 "GET", 
n=n@entry=2000,

fp=0x7fd8ffb234e0 <_IO_2_1_stdin_>) at iofgets.c:56
#6  0x0041727e in cgi_handle_http_request (zIpAddr=out>, zIpAddr@entry=0x0) at ./src/cgi.c:1414

#7  0x0044f9c1 in cmd_webserver () at ./src/main.c:2511
#8  0x00407fec in main (argc=, argv=out>) at ./src/main.c:760


(gdb) bt
#0  0x7fd8ff858ba0 in __read_nocancel () at 
../sysdeps/unix/syscall-template.S:81
#1  0x7fd8ff7f23a0 in _IO_new_file_underflow (fp=0x7fd8ffb234e0 
<_IO_2_1_stdin_>) at fileops.c:605
#2  0x7fd8ff7f31ce in __GI__IO_default_uflow (fp=0x7fd8ffb234e0 
<_IO_2_1_stdin_>) at genops.c:435
#3  0x7fd8ff7e7f84 in __GI__IO_getline_info 
(fp=fp@entry=0x7fd8ffb234e0 <_IO_2_1_stdin_>,
buf=buf@entry=0x7ffd56077bf0 "ut", n=n@entry=1999, 
delim=delim@entry=10, extract_delim=extract_delim@entry=1,

eof=eof@entry=0x0) at iogetline.c:69
#4  0x7fd8ff7e8078 in __GI__IO_getline (fp=fp@entry=0x7fd8ffb234e0 
<_IO_2_1_stdin_>,
buf=buf@entry=0x7ffd56077bf0 "ut", n=n@entry=1999, 
delim=delim@entry=10, extract_delim=extract_delim@entry=1)

at iogetline.c:38
#5  0x7fd8ff7e6e96 in _IO_fgets (buf=buf@entry=0x7ffd56077bf0 "ut", 
n=n@entry=2000,

fp=0x7fd8ffb234e0 <_IO_2_1_stdin_>) at iofgets.c:56
#6  0x00417118 in cgi_handle_http_request 
(zIpAddr=zIpAddr@entry=0x0) at ./src/cgi.c:1376

#7  0x0044f9c1 in cmd_webserver () at ./src/main.c:2511
#8  0x00407fec in main (argc=, argv=out>) at ./src/main.c:760


(gdb) bt
#0  0x7fd8ff858ba0 in __read_nocancel () at 
../sysdeps/unix/syscall-template.S:81
#1  0x7fd8ff7f23a0 in _IO_new_file_underflow (fp=0x7fd8ffb234e0 
<_IO_2_1_stdin_>) at fileops.c:605
#2  0x7fd8ff7f31ce in __GI__IO_default_uflow (fp=0x7fd8ffb234e0 
<_IO_2_1_stdin_>) at genops.c:435
#3  0x7fd8ff7e7f84 in __GI__IO_getline_info 
(fp=fp@entry=0x7fd8ffb234e0 <_IO_2_1_stdin_>,
buf=buf@entry=0x7ffd56077bf0 "ut", n=n@entry=1999, 
delim=delim@entry=10, extract_delim=extract_delim@entry=1,

eof=eof@entry=0x0) at iogetline.c:69
#4  0x7fd8ff7e8078 in __GI__IO_getline (fp=fp@entry=0x7fd8ffb234e0 
<_IO_2_1_stdin_>,
buf=buf@entry=0x7ffd56077bf0 "ut", n=n@entry=1999, 
delim=delim@entry=10, extract_delim=extract_delim@entry=1)

at iogetline.c:38
#5  0x7fd8ff7e6e96 in _IO_fgets (buf=buf@entry=0x7ffd56077bf0 "ut", 
n=n@entry=2000,

fp=0x7fd8ffb234e0 <_IO_2_1_stdin_>) at iofgets.c:56
#6  0x00417118 in cgi_handle_http_request 
(zIpAddr=zIpAddr@entry=0x0) at ./src/cgi.c:1376

#7  0x0044f9c1 in cmd_webserver () at ./src/main.c:2511
#8  0x00407fec in main (argc=, argv=out>) at ./src/main.c:760


(gdb) bt
#0  0x7fd8ff858ba0 in __read_nocancel () at 
../sysdeps/unix/syscall-template.S:81
#1  

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 18, 2017, at 7:04 PM, bch  wrote:
> 
> Does contemporary Linux not randomize its PIDs?

It may be an option, but it isn’t happening on a near-stock CentOS 7 box I have 
near at hand here:

$ ls > /dev/null & echo $! 

Run that repeatedly on a quiet system, and chances are, you’ll get steadily 
increasing PID. Any gaps you get would be other processes starting in the 
background.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[bch]

On Mon, Nov 6, 2017 at 9:19 AM Warren Young  wrote:

> On Nov 3, 2017, at 12:08 PM, Richard Hipp  wrote:
> >
> > On 11/3/17, Olivier R.  wrote:
> >>
> >>
> >> Sorry. My knowledge of the C toolchain is null.
> >
> > The next step will be to figure out
> > how to attach the debugger to a hung process.
>
> Problem #1 could be fixed (in principle) without any more help from you,
> Oliver: PIDs 888 and 893 are zombies, meaning Fossil is forking off
> children without calling wait() on them.  That’s why their VIRT column
> shows as 0 in your screenshot: the kernel has stripped all resources from
> them it can, and is holding onto only the exit status and such for the
> parent’s benefit.  This is a bug in Fossil, plain and simple.
>
> That said, zombies are nearly harmless, merely adding noise to the process
> table.  They don’t explain your actual symptom.
>
> The remaining PIDs are all certainly a single parent with multiple
> children.  You’d have to run top in “tree” mode or show the PPID column to
> find out which one is the parent.  You can tell without doing that by the
> fact that all of the VIRT column values are identical, meaning that within
> the limits of top’s reporting resolution, the children are allocating no
> dynamic virtual memory of their own, which is what we’d expect from a
> forking HTTP child-per-conn model.
>
> Given all of that, I’d just pick one of the PIDs and attach to it:
>
> $ gdb -p 26819
>
> If that works, say “bt” when attached, then “quit” to detach again.  Post
> the backtrace output here, Oliver.
>
> If it doesn’t work, it’s probably due to lack of debugging permission on
> the target system, in which case you’ve got some sysadminning ahead of you,
> not on topic here.
>
> But, this does not look like a madly-spinning system.  The CPU is idle and
> the PIDs are pretty far apart.
>


Does contemporary Linux not randomize its PIDs?


> Basically, it’s looking like each one is the result of an HTTP transaction
> and the child just isn’t dying at transaction end as it should.  This
> should only be a serious problem when the children collectively hold so
> many resources that the system can’t run properly.
>
> Bottom line, I don’t think the top output explains the problem.
> ___
> fossil-users mailing list
> fossil-users@lists.fossil-scm.org
> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
>
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Dec 18, 2017, at 6:52 AM, Olivier R.  wrote:
> 
> When gdb was active, Fossil didn’t answer when asking for a webpage. It 
> seemed blocked.

That’s exactly what happens.  If you want the process to run while GDB remains 
attached, say “cont” after attaching.

> Here is what gdb said:

Did it say the same thing if you reattached each time?

What does “fossil ver” say?  I ask because line numbers are more helpful if you 
also give the checkin ID they refer to.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Olivier R.]

OK. It happened again while I was away for few days.

The main process has created 10 subprocesses. Fossil was very slow.

I used gdb on the main process.

When gdb was active, Fossil didn’t answer when asking for a webpage. It 
seemed blocked. And Fossil was responsive again few seconds after I quit 
gdb.


This time I didn’t kill the processes. I can try to do something else if 
you want.


Here is what gdb said:

---
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 17876
Reading symbols from /usr/bin/fossil2...done.
Reading symbols from /usr/local/lib/libz.so.1...(no debugging symbols 
found)...done.

Loaded symbols for /usr/local/lib/libz.so.1
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols 
from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.19.so...done.

done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols 
from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.19.so...done.

done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols 
from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.19.so...done.

done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from 
/usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done.

done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x7fd8ff85e873 in __select_nocancel () at 
../sysdeps/unix/syscall-template.S:81

81  ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0  0x7fd8ff85e873 in __select_nocancel () at 
../sysdeps/unix/syscall-template.S:81
#1  0x00416162 in cgi_http_server (mnPort=mnPort@entry=8080, 
mxPort=mxPort@entry=8180, zBrowser=, zBrowser@entry=0x0,

zIpAddr=zIpAddr@entry=0x0, flags=flags@entry=12) at ./src/cgi.c:1845
#2  0x0044f92a in cmd_webserver () at ./src/main.c:2493
#3  0x00407fec in main (argc=, argv=out>) at ./src/main.c:760

(gdb) quit
A debugging session is active.

Inferior 1 [process 17876] will be detached.

Quit anyway? (y or n) y




Le 06/11/2017 à 18:19, Warren Young a écrit :


Problem #1 could be fixed (in principle) without any more help from
you, Oliver: PIDs 888 and 893 are zombies, meaning Fossil is forking
off children without calling wait() on them.  That’s why their VIRT
column shows as 0 in your screenshot: the kernel has stripped all
resources from them it can, and is holding onto only the exit status
and such for the parent’s benefit.  This is a bug in Fossil, plain
and simple.

That said, zombies are nearly harmless, merely adding noise to the
process table.  They don’t explain your actual symptom.

The remaining PIDs are all certainly a single parent with multiple
children.  You’d have to run top in “tree” mode or show the PPID
column to find out which one is the parent.  You can tell without
doing that by the fact that all of the VIRT column values are
identical, meaning that within the limits of top’s reporting
resolution, the children are allocating no dynamic virtual memory of
their own, which is what we’d expect from a forking HTTP
child-per-conn model.

Given all of that, I’d just pick one of the PIDs and attach to it:

$ gdb -p 26819

If that works, say “bt” when attached, then “quit” to detach again.
Post the backtrace output here, Oliver.

If it doesn’t work, it’s probably due to lack of debugging permission
on the target system, in which case you’ve got some sysadminning
ahead of you, not on topic here.

But, this does not look like a madly-spinning system.  The CPU is
idle and the PIDs are pretty far apart.

Basically, it’s looking like each one is the result of an HTTP
transaction and the child just isn’t dying at transaction end as it
should.  This should only be a serious problem when the children
collectively hold so many resources that the system can’t run
properly.


___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Olivier R.]

Le 06/11/2017 à 18:19, Warren Young a écrit :


The remaining PIDs are all certainly a single parent with multiple
children.  You’d have to run top in “tree” mode or show the PPID
column to find out which one is the parent.  You can tell without
doing that by the fact that all of the VIRT column values are
identical, meaning that within the limits of top’s reporting
resolution, the children are allocating no dynamic virtual memory of
their own, which is what we’d expect from a forking HTTP
child-per-conn model.

Given all of that, I’d just pick one of the PIDs and attach to it:

$ gdb -p 26819

If that works, say “bt” when attached, then “quit” to detach again.
Post the backtrace output here, Oliver.

If it doesn’t work, it’s probably due to lack of debugging permission
on the target system, in which case you’ve got some sysadminning
ahead of you, not on topic here.

But, this does not look like a madly-spinning system.  The CPU is
idle and the PIDs are pretty far apart.

Basically, it’s looking like each one is the result of an HTTP
transaction and the child just isn’t dying at transaction end as it
should.  This should only be a serious problem when the children
collectively hold so many resources that the system can’t run
properly.

Bottom line, I don’t think the top output explains the problem. 


Thank you for the explanation.

The server on which Fossil is running is dedicated to this task, I don’t 
do anything else with it.


When I reported the problem I had killed all processes and had 
relaunched Fossil. I now run the version I compiled, and will do as you 
said as soon as I noticed the problem occurs again. I still have no clue 
of what can produce this behavior.


We’ll have to wait.


Olivier
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Warren Young]

On Nov 3, 2017, at 12:08 PM, Richard Hipp  wrote:
> 
> On 11/3/17, Olivier R.  wrote:
>> 
>> 
>> Sorry. My knowledge of the C toolchain is null.
> 
> The next step will be to figure out
> how to attach the debugger to a hung process.

Problem #1 could be fixed (in principle) without any more help from you, 
Oliver: PIDs 888 and 893 are zombies, meaning Fossil is forking off children 
without calling wait() on them.  That’s why their VIRT column shows as 0 in 
your screenshot: the kernel has stripped all resources from them it can, and is 
holding onto only the exit status and such for the parent’s benefit.  This is a 
bug in Fossil, plain and simple.

That said, zombies are nearly harmless, merely adding noise to the process 
table.  They don’t explain your actual symptom.

The remaining PIDs are all certainly a single parent with multiple children.  
You’d have to run top in “tree” mode or show the PPID column to find out which 
one is the parent.  You can tell without doing that by the fact that all of the 
VIRT column values are identical, meaning that within the limits of top’s 
reporting resolution, the children are allocating no dynamic virtual memory of 
their own, which is what we’d expect from a forking HTTP child-per-conn model.

Given all of that, I’d just pick one of the PIDs and attach to it:

$ gdb -p 26819

If that works, say “bt” when attached, then “quit” to detach again.  Post the 
backtrace output here, Oliver.

If it doesn’t work, it’s probably due to lack of debugging permission on the 
target system, in which case you’ve got some sysadminning ahead of you, not on 
topic here.

But, this does not look like a madly-spinning system.  The CPU is idle and the 
PIDs are pretty far apart.

Basically, it’s looking like each one is the result of an HTTP transaction and 
the child just isn’t dying at transaction end as it should.  This should only 
be a serious problem when the children collectively hold so many resources that 
the system can’t run properly.

Bottom line, I don’t think the top output explains the problem.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Richard Hipp]

On 11/3/17, Olivier R.  wrote:
> Le 03/11/2017 à 10:22, Richard Hipp a écrit :
>> On 11/3/17, Olivier R.  wrote:
>>>
>>> And today, Fossil was already very slow and I discovered that there were
>>> more than ten processes of Fossil running.
>>
>> Can you compile with symbols enabled ("-g") and then attach to one of
>> the hung processes using a debugger to see what it is doing?
>
> Sorry. My knowledge of the C toolchain is null.

Looks like the default Makefile includes -g automatically.  So there
is nothing for you to do there.  The next step will be to figure out
how to attach the debugger to a hung process.  Maybe somebody else on
this list can give you detailed instructions, since I'm not really
sure myself.

-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Olivier R.]

Le 03/11/2017 à 10:22, Richard Hipp a écrit :

On 11/3/17, Olivier R.  wrote:


And today, Fossil was already very slow and I discovered that there were
more than ten processes of Fossil running.


Can you compile with symbols enabled ("-g") and then attach to one of
the hung processes using a debugger to see what it is doing?


Sorry. My knowledge of the C toolchain is null.

I installed the required packages.
I compiled and make install what I found in compat/zlib.

Then I used the command:

./configure --with-openssl=none; make

(I didn’t succeed to compile with openssl even when I tell it where it 
is. I probably did it wrong.)


But I don’t know where I should set the -g option.
And I have no clue what to do about the step 2 with “a debugger”.

Maybe I should first update to Fossil 2.4 and see if the problem still 
occurs.



Olivier
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] [Bug?] [server] Processes of Fossil popping up unexpectedly

[Richard Hipp]

On 11/3/17, Olivier R.  wrote:
>
> And today, Fossil was already very slow and I discovered that there were
> more than ten processes of Fossil running.

Can you compile with symbols enabled ("-g") and then attach to one of
the hung processes using a debugger to see what it is doing?
-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users