I'm monitoring whole server loadavg and ploting this data as graphs.
us+ni (cpu usage) as orange, iowait (wa) as red and sys (sy) as blue on graphs:
http://s15.postimg.org/xdor84ybv/image.png
http://s16.postimg.org/4ytjw8blh/image.png
i've marked assp restart with black arrow. loadavg goes immeadetly
down and stays low for some time.
When I restart assp on other servers loadavg never goes down. Also
assp is not showing 100% cpu ussage in top there (and overall cpu time
is much much lower).
To monitor assp I use php fsockopen() with a timeout of 15 seconds.
Only on this server (of of many more I run) assp gets pedroidcaly
detected as not responding in 15s.
2015-01-28 11:15 GMT+01:00 Colin <colin.war...@gmail.com>:
> How are you monitoring the usage? I remember getting caught out by what
> you are describing when I first started monitoring performance.
>
> Don't forget that top by default shows you the performance figures
> relating to a single core. If you have a multiple core system then you
> will regularly see more than 100% and a load higher than 1. Useful
> options are I (capital i) to change the CPU% to relative to the total
> and push f and select the P column to show which CPU/core each process
> is running on.
>
> I also saw what you were seeing in strace as natural behaviour. What you
> should find is that the WAKE events are preceeded by a poll event doing
> POLLIN. If there aren't any connections waiting it will return resource
> temporarily unavailable and it will go back to WAIT.
>
> I do see a lot of gettimeofday in strace too and I think I get a couple
> of minutes delay in things being written to the logs which is odd.
>
> What monitoring service do you have and how does it identify ASSP as not
> responding?
>
> All the best,
> Colin.
>
> On 28/01/2015 09:51, krz...@gmail.com wrote:
>> I have assp (ASSP version 2.4.1(14132)) running on multiple servers.
>> Those servers have exacly same configuration, os (mirrored), hardware.
>> On one of these servers assp is causing high cpu ussage. After
>> restarting assp cpu usage is rising constantly and "slowly". After
>> about 4 hours it it 100% on i7-4770. After about 8-16 hours assp is so
>> slow to respond that monitoring service sees it as not running and
>> restarts it. Server has low traffic volume, much lower that most of my
>> other servers. There is nothing in assp logs (even on highest
>> verbosity for connection logging) - a smtp session every 5-30 seconds
>> maybe. Assp "ASSP Worker/DB/Regex Status" shows workers in
>> "ThreadGetNewCon" status. Strace executed on assp pid shows a lot more
>> of FUTEX_WAKE_OP_PRIVATE in comparison than on assp on other servers.
>> There is no problems with other software on server and nothing on
>> dmesg so I don't think it is a hardware problem. I even tried
>> reinstalling fresh pel 5.18 with new modules by mod_inst.pl. Can
>> anyone help?
>>
>> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
>> sched_yield() = 0
>> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
>> sched_yield() = 0
>> sched_yield() = 0
>> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x7fb2da74fa20, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN
>> (Resource temporarily unavailable)
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
>> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x7fb2da74fa20, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN
>> (Resource temporarily unavailable)
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
>> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
>> sched_yield() = 0
>> futex(0x7fb2da74fa5c, FUTEX_WAIT_PRIVATE, 88147493, NULL) = -1 EAGAIN
>> (Resource temporarily unavailable)
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
>> futex(0x7fb2da74fa58, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x7fb2da74fa5c, FUTEX_WAIT_PRIVATE, 88147495, NULL) = -1 EAGAIN
>> (Resource temporarily unavailable)
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
>> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
>> futex(0x7fb2da74fa58, FUTEX_WAKE_PRIVATE, 1) = 1
>> futex(0x7fb2da74fa5c, FUTEX_WAIT_PRIVATE, 88147501, NULL) = -1 EAGAIN
>> (Resource temporarily unavailable)
>>
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming. The Go Parallel Website,
>> sponsored by Intel and developed in partnership with Slashdot Media, is your
>> hub for all things parallel software development, from weekly thought
>> leadership blogs to news, videos, case studies, tutorials and more. Take a
>> look and join the conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
