Re: [naviserver-devel] Long Accept Times

[David Osborne]

Thank you very much for the detailed info.

Can I check something...

I'm understanding that runtimes are dependent on the client in cases where
there are no writer threads.
Given that the smallest writersize available is 1024, it is impossible
(regardless of whether it's desirable) to have all requests serviced by
writer threads since small requests will always be delivered by the
connection thread.

Therefore, it is fair to say, that it is impossible to completely eliminate
the effect network delivery times has on cumulative runtimes?


On 7 March 2018 at 18:21, Gustaf Neumann  wrote:

> On 3/7/18 5:28 PM, David Osborne wrote:
>
> Thank you very much for that info Gustaf.
> We'll have a look at whether we can adapt the ideas in boomerang to our
> environment.
>
> Yes we suspected the Accept timings might be outwith the control of the
> server.
> Am I right in thinking that Runtimes (if not using writer threads) are
> also affected by network conditions since they include the time taken to
> deliver content back to the client?
>
> Here are a few images for visualizing what happens: a typical request has
> a receiving, processing and delivery phase:
>
> if NaviServer processes a request without "read-ahead", and there are no
> writer threads involved, the full time is spent in a connection thread, all
> is accounted in the "runtime". The time of the "receive" and "delivery"
> phase is only under partial control of the server; a slow sending or
> receiving client can cause long blocking time of a connection thread, which
> is a limited resource. This can be as well the cause of potential "slow
> read/write" attacks to servers.
>
> Fortunately, NaviServer offers the means to delegate the receive and
> delivery phases to separate threads using async i/o, where every of these
> threads is able to handle thousands of sockets concurrently. The "driver"
> and "spooling" threads of NaviServer handle the receiving part, and the
> "writer" threads the delivery part. The http/tls drivers are configured
> with "read-ahead", other drivers are not, so that the receiving part is
> just the accept operation. With "readahead", one can define multiple driver
> threads listening on one port, and 0..n spooling threads for huge (file)
> uploads.
>
> With these i/o threads in place, the picture looks like the following for
> read-ahead threads:
>
> where only the orange time spans are tunes spent in a connection thread.
> With these in i/o threads every connection thread is able to handle much
> more requests per second than without those.  For application development,
> the times in the connection threads are the one which are influenced
> primarily by the application developers, so these are reported as runtimes
> of the requests. In reality, the picture is more complicated, since the
> server sends data back as soon it is available (it does not wait, until the
> whole reply is computed), such that the delivery starts actually already in
> the the "orange" phase, 2 threads can work pipelined on one one request.
> When writer threads are enabled, every writer operation is essentially just
> an enqueuing operation. When there are no writer threads enabled, the
> runtime will be dependent on the client (and the connection quality and
> conditions on the way to the client). So, writer threads are highly
> recommended.
>
> hope this helps, all the best
>
> -g
>
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> naviserver-devel mailing list
> naviserver-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/naviserver-devel
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Long Accept Times

[Gustaf Neumann]

On 3/7/18 5:28 PM, David Osborne wrote:

Thank you very much for that info Gustaf.
We'll have a look at whether we can adapt the ideas in boomerang to 
our environment.


Yes we suspected the Accept timings might be outwith the control of 
the server.
Am I right in thinking that Runtimes (if not using writer threads) are 
also affected by network conditions since they include the time taken 
to deliver content back to the client?


Here are a few images for visualizing what happens: a typical request 
has a receiving, processing and delivery phase:


if NaviServer processes a request without "read-ahead", and there are no 
writer threads involved, the full time is spent in a connection thread, 
all is accounted in the "runtime". The time of the "receive" and 
"delivery" phase is only under partial control of the server; a slow 
sending or receiving client can cause long blocking time of a connection 
thread, which is a limited resource. This can be as well the cause of 
potential "slow read/write" attacks to servers.


Fortunately, NaviServer offers the means to delegate the receive and 
delivery phases to separate threads using async i/o, where every of 
these threads is able to handle thousands of sockets concurrently. The 
"driver" and "spooling" threads of NaviServer handle the receiving part, 
and the "writer" threads the delivery part. The http/tls drivers are 
configured with "read-ahead", other drivers are not, so that the 
receiving part is just the accept operation. With "readahead", one can 
define multiple driver threads listening on one port, and 0..n spooling 
threads for huge (file) uploads.


With these i/o threads in place, the picture looks like the following 
for read-ahead threads:


where only the orange time spans are tunes spent in a connection thread. 
With these in i/o threads every connection thread is able to handle much 
more requests per second than without those.  For application 
development, the times in the connection threads are the one which are 
influenced primarily by the application developers, so these are 
reported as runtimes of the requests. In reality, the picture is more 
complicated, since the server sends data back as soon it is available 
(it does not wait, until the whole reply is computed), such that the 
delivery starts actually already in the the "orange" phase, 2 threads 
can work pipelined on one one request. When writer threads are enabled, 
every writer operation is essentially just an enqueuing operation. When 
there are no writer threads enabled, the runtime will be dependent on 
the client (and the connection quality and conditions on the way to the 
client). So, writer threads are highly recommended.


hope this helps, all the best

-g

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Long Accept Times

[David Osborne]

Thank you very much for that info Gustaf.
We'll have a look at whether we can adapt the ideas in boomerang to our
environment.

Yes we suspected the Accept timings might be outwith the control of the
server.
Am I right in thinking that Runtimes (if not using writer threads) are also
affected by network conditions since they include the time taken to deliver
content back to the client?

Thanks again,
--
David

On 5 March 2018 at 20:25, Gustaf Neumann  wrote:

> Dear David,
>
> i would not recommend to use the accept times are a measure of
> performance, since it measures the time between the accept of an incoming
> connection until the time, when the request is added to the queue, since
> for one "accept", there might be multiple requests added to the queue (on
> HTTP persistent connections).
>
> It is true, that this time contains for drivers with read-ahead configured
> (standard case) the time for receiving the request from the client. But
> note, that this time depends on the connection quality of the client. A
> client on a remote location connected via smart phone will lead to much
> worse values than a user on the local network. Increasing the bandwidth to
> your Internet provider might not change this picture. But note, that this
> high latency does not hit the server, since reading is performed with async
> i/o. Similarly, largish post requests will result in larger accept times.
>
> The interpretation of the accepttime is not easy, and depends also on the
> overall system architecture, e.g. whether or not NaviServer is running
> behind a proxy server. See below for two different systems: the first chart
> show the year statistics from a system running without a reverse proxy in
> front of it, the second chart is from a system is running behind nginx. The
> first system show high accept times (average 1.1 seconds), the second one
> shows low accept times (average 200 micro seconds). For the first system,
> one sees rather growing accept times (on the logarithmic scale), which says
> probably, that the number of clients with persistent connections is
> growing. On the second system, one can see a change in the accept time
> around mid January from typically below 100 microseconds (10^-4s) to the
> current value.  Interestingly, at that time, we fixed the spectre/meltdown
> vulnerabilities, and the kernel latencies increased.
>
> So, in short for some use cases, the accept-timings might be useful, but
> typically, these values can't be changed by faster soft- or hardware. We
> exclude the accept times often in our performance analysis.
>
> If you are interested in the real user experience, i would recommend a
> measurement tool like boomerang [1,2]. One lesson to learn from that is
> that only a relatively small fraction of the perceived performance is due
> direct server performance (in the example of openacs.org, total server
> runtime is 36ms, while the performance is around 2 secs; you can get from
> boomerang as well the "request time", which is the time after the TCP
> connect has finished until the request is received at the server)
>
> all the best
> -g
>
> [1] https://openacs.org/forums/message-view?message_id=5387444
> [2] https://github.com/openacs/boomerang
>
>
> [image: yearly graph]
>
>
>
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel

Re: [naviserver-devel] Long Accept Times

[Gustaf Neumann]

Dear David,

i would not recommend to use the accept times are a measure of 
performance, since it measures the time between the accept of an 
incoming connection until the time, when the request is added to the 
queue, since for one "accept", there might be multiple requests added to 
the queue (on HTTP persistent connections).


It is true, that this time contains for drivers with read-ahead 
configured (standard case) the time for receiving the request from the 
client. But note, that this time depends on the connection quality of 
the client. A client on a remote location connected via smart phone will 
lead to much worse values than a user on the local network. Increasing 
the bandwidth to your Internet provider might not change this picture. 
But note, that this high latency does not hit the server, since reading 
is performed with async i/o. Similarly, largish post requests will 
result in larger accept times.


The interpretation of the accepttime is not easy, and depends also on 
the overall system architecture, e.g. whether or not NaviServer is 
running behind a proxy server. See below for two different systems: the 
first chart show the year statistics from a system running without a 
reverse proxy in front of it, the second chart is from a system is 
running behind nginx. The first system show high accept times (average 
1.1 seconds), the second one shows low accept times (average 200 micro 
seconds). For the first system, one sees rather growing accept times (on 
the logarithmic scale), which says probably, that the number of clients 
with persistent connections is growing. On the second system, one can 
see a change in the accept time around mid January from typically below 
100 microseconds (10^-4s) to the current value.  Interestingly, at that 
time, we fixed the spectre/meltdown vulnerabilities, and the kernel 
latencies increased.


So, in short for some use cases, the accept-timings might be useful, but 
typically, these values can't be changed by faster soft- or hardware. We 
exclude the accept times often in our performance analysis.


If you are interested in the real user experience, i would recommend a 
measurement tool like boomerang [1,2]. One lesson to learn from that is 
that only a relatively small fraction of the perceived performance is 
due direct server performance (in the example of openacs.org, total 
server runtime is 36ms, while the performance is around 2 secs; you can 
get from boomerang as well the "request time", which is the time after 
the TCP connect has finished until the request is received at the server)


all the best
-g

[1] https://openacs.org/forums/message-view?message_id=5387444
[2] https://github.com/openacs/boomerang


yearly graph


On 3/5/18 5:31 PM, David Osborne wrote:

Hi,

One of our installations is appearing to randomly experience long 
accept times.


While all other requests are flowing nicely, suddenly one request will 
have an accept time of between 2-20 seconds (as measured in the access 
log with logpartialtimes).


Our understanding is that this is the time it takes for the driver to 
receive the entire request from the client, and long accept times can 
be due to clients connecting with very low bandwidths.


However, we're not really seeing any correlation between slow times 
and any particular clients or request type. Plus there doesn't seem to 
be any correlation to CPU/network/disk load.


I was wondering if there are other factors apart from bandwidth we 
should be looking at which could be affecting accept times?


Regards,
--
David


--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
naviserver-devel mailing list
naviserver-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/naviserver-devel