How are you monitoring the usage? I remember getting caught out by what
you are describing when I first started monitoring performance.
Don't forget that top by default shows you the performance figures
relating to a single core. If you have a multiple core system then you
will regularly see more than 100% and a load higher than 1. Useful
options are I (capital i) to change the CPU% to relative to the total
and push f and select the P column to show which CPU/core each process
is running on.
I also saw what you were seeing in strace as natural behaviour. What you
should find is that the WAKE events are preceeded by a poll event doing
POLLIN. If there aren't any connections waiting it will return resource
temporarily unavailable and it will go back to WAIT.
I do see a lot of gettimeofday in strace too and I think I get a couple
of minutes delay in things being written to the logs which is odd.
What monitoring service do you have and how does it identify ASSP as not
responding?
All the best,
Colin.
On 28/01/2015 09:51, krz...@gmail.com wrote:
> I have assp (ASSP version 2.4.1(14132)) running on multiple servers.
> Those servers have exacly same configuration, os (mirrored), hardware.
> On one of these servers assp is causing high cpu ussage. After
> restarting assp cpu usage is rising constantly and "slowly". After
> about 4 hours it it 100% on i7-4770. After about 8-16 hours assp is so
> slow to respond that monitoring service sees it as not running and
> restarts it. Server has low traffic volume, much lower that most of my
> other servers. There is nothing in assp logs (even on highest
> verbosity for connection logging) - a smtp session every 5-30 seconds
> maybe. Assp "ASSP Worker/DB/Regex Status" shows workers in
> "ThreadGetNewCon" status. Strace executed on assp pid shows a lot more
> of FUTEX_WAKE_OP_PRIVATE in comparison than on assp on other servers.
> There is no problems with other software on server and nothing on
> dmesg so I don't think it is a hardware problem. I even tried
> reinstalling fresh pel 5.18 with new modules by mod_inst.pl. Can
> anyone help?
>
> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
> sched_yield() = 0
> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
> sched_yield() = 0
> sched_yield() = 0
> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x7fb2da74fa20, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN
> (Resource temporarily unavailable)
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 2
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x7fb2da74fa20, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN
> (Resource temporarily unavailable)
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
> sched_yield() = 0
> futex(0x7fb2da74fa5c, FUTEX_WAIT_PRIVATE, 88147493, NULL) = -1 EAGAIN
> (Resource temporarily unavailable)
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7fb2da74fa58, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x7fb2da74fa5c, FUTEX_WAIT_PRIVATE, 88147495, NULL) = -1 EAGAIN
> (Resource temporarily unavailable)
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x7fb2da74fa5c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x7fb2da74fa58,
> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x7fb2da74fa20, FUTEX_WAKE_PRIVATE, 1) = 0
> futex(0x7fb2da74fa58, FUTEX_WAKE_PRIVATE, 1) = 1
> futex(0x7fb2da74fa5c, FUTEX_WAIT_PRIVATE, 88147501, NULL) = -1 EAGAIN
> (Resource temporarily unavailable)
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming. The Go Parallel Website,
> sponsored by Intel and developed in partnership with Slashdot Media, is your
> hub for all things parallel software development, from weekly thought
> leadership blogs to news, videos, case studies, tutorials and more. Take a
> look and join the conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
------------------------------------------------------------------------------
Dive into the World of Parallel Programming. The Go Parallel Website,
sponsored by Intel and developed in partnership with Slashdot Media, is your
hub for all things parallel software development, from weekly thought
leadership blogs to news, videos, case studies, tutorials and more. Take a
look and join the conversation now. http://goparallel.sourceforge.net/
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
