Re: [fossil-users] Why Fossil is so fast?

2018-04-02 Thread Svyatoslav Mishyn 
(Sat, 31 Mar 08:22) Richard Hipp:
> On 3/31/18, Svyatoslav Mishyn  wrote:
> > "max-loadavg" setting can't help as it does not affect /vdiff pages.
> 
> We could fix that.  Should we?

then /fdiff, /vpatch, /info pages should probably also be included
if "max-loadavg" effect will be eventually expanded.


But for now, for me it is OK, to just limit number of
requests/connections via nginx itself
(and then if needed block IPs via fail2ban -> iptables).


-- 
https://www.juef.space/
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Why Fossil is so fast?

2018-03-31 Thread Richard Hipp 
On 3/31/18, Svyatoslav Mishyn  wrote:
> "max-loadavg" setting can't help as it does not affect /vdiff pages.
>

We could fix that.  Should we?
-- 
D. Richard Hipp
d...@sqlite.org
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Why Fossil is so fast?

2018-03-31 Thread Svyatoslav Mishyn 
(Mon, 26 Mar 16:40) Warren Young:
> On Mar 26, 2018, at 2:45 PM, Warren Young  wrote:
> > 
> > On Mar 26, 2018, at 2:15 PM, Svyatoslav Mishyn 
> >  wrote:
> >> 
> >> Here are results of r.sh when stress.sh was run (and all RAM was used
> >> on VPS):
> 
> I’ve thought a bit more about this stress.sh script.  It is based on ab, 
> which I presume is the Apache Benchmark program.  You aren’t giving it -C, 
> which means it’s just bouncing off that URL and sending you back to the login 
> page on each HTTP hit.  Thus, it is not at all like a real user trying to use 
> the fossil-scm.org repository remotely.
> 
> Monitor your HTTP traffic to the Fossil server, and I think you’ll see that 
> you aren’t actually pulling vdiffs with this test.

Actually, Apache Benchmark pulls diffs without "-C" option as the
"nobody" user got "gjorz" permissions.

If I remove "o" (Check-Out) capability, then yes, will be a redirect to
/login page.


On the other hand, how to protect a VPS against such requests?
Without removing current functionality for non-logged ("nobody") users, i.e.
keep "o" capability.

"max-loadavg" setting can't help as it does not affect /vdiff pages.

Only by limiting requests by nginx to fossil.?


-- 
https://www.juef.space/
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Why Fossil is so fast?

2018-03-26 Thread Warren Young 
On Mar 26, 2018, at 2:45 PM, Warren Young  wrote:
> 
> On Mar 26, 2018, at 2:15 PM, Svyatoslav Mishyn  
> wrote:
>> 
>> Here are results of r.sh when stress.sh was run (and all RAM was used
>> on VPS):

I’ve thought a bit more about this stress.sh script.  It is based on ab, which 
I presume is the Apache Benchmark program.  You aren’t giving it -C, which 
means it’s just bouncing off that URL and sending you back to the login page on 
each HTTP hit.  Thus, it is not at all like a real user trying to use the 
fossil-scm.org repository remotely.

Monitor your HTTP traffic to the Fossil server, and I think you’ll see that you 
aren’t actually pulling vdiffs with this test.

> It is bad science to change more than one variable at a time.  You’ve got a 
> multiply confounded result set here.

It might help to list the confounding factors:

1. Different network path between the two servers under test.

2. Different hardware for each server.

3. Probably different VPS technology on each host.

4. Different repository?  (It’s not clear to me whether you’re working with a 
clone of the same repo on both sides.)

5. Real users vs ab(1) faux users.

You can’t isolate causes until you control for each of these variables.

On top of that, you’re taking Fossil’s reported time deltas as if they’re 
high-quality benchmark data, when I’ve already shown that the numbers are 
biased by +1ms, which is on the order of the time differences you’re working 
with.  You need to be getting your timing information from a purer source.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Why Fossil is so fast?

2018-03-26 Thread Warren Young 
On Mar 26, 2018, at 2:15 PM, Svyatoslav Mishyn  
wrote:
> 
> Here are results of r.sh when stress.sh was run (and all RAM was used
> on VPS):
> 2018-03-26-19:34:08 time generation: 0.001s; load average: 10.909180
> 2018-03-26-19:34:11 time generation: 0.001s; load average: 10.909180
> 2018-03-26-19:34:14 time generation: 0.001s; load average: 12.918450
> 2018-03-26-19:34:15 time generation: 0.001s; load average: 12.918450
> 2018-03-26-19:34:18 time generation: 0.001s; load average: 14.767090
> 2018-03-26-19:34:22 time generation: 0.001s; load average: 14.767090
> 2018-03-26-19:34:24 time generation: 0.001s; load average: 16.547850
> 
> and against fossil-scm.org itself (only r.sh):
> 2018-03-26-20:01:03 time generation: 0.003s; load average: 0.11
> 2018-03-26-20:01:07 time generation: 0.004s; load average: 0.10
> 2018-03-26-20:01:13 time generation: 0.003s; load average: 0.09
> 2018-03-26-20:01:20 time generation: 0.003s; load average: 0.08
> 2018-03-26-20:01:26 time generation: 0.003s; load average: 0.07
> 2018-03-26-20:01:29 time generation: 0.003s; load average: 0.07
> 2018-03-26-20:01:32 time generation: 0.003s; load average: 0.06

I don’t see that you’ve factored out the TCP connection round-trip time.  Even 
from one datacenter to another datacenter on the same continent across a fat 
Internet backbone, a TCP connection still takes at least half a millisecond.  
Across continents, it becomes tens or hundreds of ms, and when you involve the 
residential ISPs, it can be even longer.

It is bad science to change more than one variable at a time.  You’ve got a 
multiply confounded result set here.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Why Fossil is so fast?

2018-03-26 Thread Svyatoslav Mishyn 
(Mon, 26 Mar 09:50) Warren Young:
> > I compared page generation speed with some other Fossil repositories:
> > Fossil itself, SQLite, Tcl
> > using the same pages which are not strictly related to
> > repository size and age: like
> > /help, /sitemap, /stat, /wiki, /version?verbose
> > 
> > and on these repos I got various timings after refreshing page,
> > while on mine always 0.001s
> 
> You are presumably using a new or at least young repository, thus the various 
> log(N) operations are using very small values of N, thus take very few CPU 
> cycles to execute.  Additionally, you are doing this on a system that is not 
> under load, and which has hot RAM and mass storage caches.
> 
> To this, you are comparing other repositories with large N, which probably 
> have other concurrent users at any given time, and which are likely to have 
> quite a few cache misses.
> 
> Naturally your repository is “faster”.  Put it under the same loads, and it 
> will slow down as well.

OK, thanks for the detailed answer.

But why I still got better timings even under heavy load?


I have added a temporary Fossil mirror, then run stress.sh.
(VPS bursted: all RAM, Swap and CPU were used, so I added after that
a rate limiting to nginx.)

Here are results of r.sh when stress.sh was run (and all RAM was used
on VPS):
2018-03-26-19:34:08 time generation: 0.001s; load average: 10.909180
2018-03-26-19:34:11 time generation: 0.001s; load average: 10.909180
2018-03-26-19:34:14 time generation: 0.001s; load average: 12.918450
2018-03-26-19:34:15 time generation: 0.001s; load average: 12.918450
2018-03-26-19:34:18 time generation: 0.001s; load average: 14.767090
2018-03-26-19:34:22 time generation: 0.001s; load average: 14.767090
2018-03-26-19:34:24 time generation: 0.001s; load average: 16.547850

and against fossil-scm.org itself (only r.sh):
2018-03-26-20:01:03 time generation: 0.003s; load average: 0.11
2018-03-26-20:01:07 time generation: 0.004s; load average: 0.10
2018-03-26-20:01:13 time generation: 0.003s; load average: 0.09
2018-03-26-20:01:20 time generation: 0.003s; load average: 0.08
2018-03-26-20:01:26 time generation: 0.003s; load average: 0.07
2018-03-26-20:01:29 time generation: 0.003s; load average: 0.07
2018-03-26-20:01:32 time generation: 0.003s; load average: 0.06

I tested only /wiki page.


-- 
https://www.juef.space/
#!/bin/sh

url="https://www.fossil-scm.org/index.html/wiki;
test_env_url="https://www.fossil-scm.org/index.html/test_env;

url="https://f.juef.space/mirrors/fossil/wiki;
test_env_url="https://f.juef.space/mirrors/fossil/test_env;

while true; do
t="$(curl -s "$url" \
| awk '/This page was generated/ { getline; print($1) }')"
la="$(curl -s "$test_env_url"   \
| awk '/load_average()/ { print(substr($3, 0, 8)) }')"

printf "%s time generation: %s; load average: %f\n" \
"$(date -u '+%F-%T')" "$t" "$la"
done
#!/bin/sh

url="https://f.juef.space/mirrors/fossil/vdiff?from=694e11a72e32bb45=1336c4af8a016772;

while true; do
ab -c 50 -n 50 "$url" > /dev/null
done
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Why Fossil is so fast?

2018-03-26 Thread Joerg Sonnenberger 
On Mon, Mar 26, 2018 at 09:50:22AM -0600, Warren Young wrote:
> On Mar 24, 2018, at 5:11 PM, Svyatoslav Mishyn  
> wrote:
> > 
> > just noticed that page generation of many of my repos is 0.001s.
> 
> The calculation for that is in the skin’s Footer code:
> 
> [expr {([utime]+[stime]+1000)/1000*0.001}]
> 
> In TH1, the utime and stime functions return microseconds, so + 1000
> means “add one millisecond”, so that when dividing microseconds down
> to seconds with the subsequent arithmetic, the resulting value can never be 
> lower than 0.001s.

That's a very complicated way to just add 0.001 if that is the
intention. If the goal is to round up, it would be better to add 999 :)

Joerg
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Why Fossil is so fast?

2018-03-26 Thread Warren Young 
On Mar 24, 2018, at 5:11 PM, Svyatoslav Mishyn  
wrote:
> 
> just noticed that page generation of many of my repos is 0.001s.

The calculation for that is in the skin’s Footer code:

[expr {([utime]+[stime]+1000)/1000*0.001}]

In TH1, the utime and stime functions return microseconds, so + 1000 means “add 
one millisecond”, so that when dividing microseconds down to seconds with the 
subsequent arithmetic, the resulting value can never be lower than 0.001s.

It would probably be more mathematically defensible to use + 500 instead to 
round to the nearest millisecond, but that might confuse people when people see 
that it claims to have run in “0.000s”, since it obviously did not run 
instantaneously.

Another mathematically defensible change would be to report that the page 
generation took “less than” the reported amount of seconds.

(Not “less than or equal to,” since a generation time of exactly 1000 µs would 
yield “0.002s” from the above calculation.)

> And that result is identical for many web-pages,
> only difference was for some /search request
> and some heavyweight check-in pages.

Your point?

If we back out the “round up to next millisecond” arithmetic explained above, 
an “0.001s” generation time report means the actual wall time to build the page 
was somewhere under 1ms.

On a 3 GHz CPU, a clock cycle is ~0.33ns, and many Intel CPU instructions do in 
fact take about 1 cycle.  Of those that take multiple CPU cycles, you can often 
wave way the difference with pipelining and speculative execution.  Thus, to a 
first approximation we have enough time to run about 3 *million* CPU 
instructions.

The wonder of it all is not how Fossil can manage to construct a few kilobytes 
of HTML by using millions of CPU instructions, but that we’ve ended up in a 
world where we’re uncertain about whether millions of CPU instructions are 
sufficient to accomplish the task!

> I compared page generation speed with some other Fossil repositories:
> Fossil itself, SQLite, Tcl
> using the same pages which are not strictly related to
> repository size and age: like
> /help, /sitemap, /stat, /wiki, /version?verbose
> 
> and on these repos I got various timings after refreshing page,
> while on mine always 0.001s

You are presumably using a new or at least young repository, thus the various 
log(N) operations are using very small values of N, thus take very few CPU 
cycles to execute.  Additionally, you are doing this on a system that is not 
under load, and which has hot RAM and mass storage caches.

To this, you are comparing other repositories with large N, which probably have 
other concurrent users at any given time, and which are likely to have quite a 
few cache misses.

Naturally your repository is “faster”.  Put it under the same loads, and it 
will slow down as well.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

[fossil-users] Why Fossil is so fast?

2018-03-24 Thread Svyatoslav Mishyn 
Hi,

just noticed that page generation of many of my repos is 0.001s.
And that result is identical for many web-pages,
only difference was for some /search request
and some heavyweight check-in pages.

(I looked just in footer info, which is
https://www.fossil-scm.org/index.html/artifact/7dd553d65672572e at least
for my repos.)

I compared page generation speed with some other Fossil repositories:
Fossil itself, SQLite, Tcl
using the same pages which are not strictly related to
repository size and age: like
/help, /sitemap, /stat, /wiki, /version?verbose

and on these repos I got various timings after refreshing page,
while on mine always 0.001s

Q1: Why I got such results?
(here is how fossil built: https://cfg.juef.space/artifact/0cf91e3734b0183a)
Q2: What are the most heavy-weight web-pages that are independent of
repository (i.e. do not depend on its size and age) to test such cases
more deeply?


Thanks.


-- 
https://www.juef.space/
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users