Tibor Simko <[email protected]> wrote:
>
> On Mon, 25 Feb 2013, Ferran Jorba wrote:
>> I've noticed that bibsched, in my two testing 1.1 installations, are
>> eating 40-50% CPU. I've done: bibsched purge, stop and start. I've
>> examined the output of mytop and mysqltuner, seing nothing strange. I
>> haven't seen a --verbose flag to help me in my quest.
>
> Is it bibsched monitor process itself that takes resources?
Now that I've observed it better, it is bibsched in automatic mode.
When I've stopped both bibscheds and with the test server mostly idle,
I've started one of them. When it is in manual mode, it does not eat
any resource. When in automatic mode, both bibssched and mysql start
consuming lots of CPU. From top monitor:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6356 mysql 20 0 761m 63m 3456 S 61 0.8 2175:40 /usr/sbin/mysqld
--basedir=/usr [...]
20700 ddd 20 0 75764 14m 1232 S 42 0.2 1:57.87 /usr/bin/python
/home/ddd/invenio/bin/bibsched start
As soon as I put in manual mode, both processess disappear from top or
htop. So, apparently it is querying the database too much for nothing.
> Does bibsched have lots of running/pending/archived task to deal with?
>
> $ echo "SELECT COUNT(*) FROM schTASK" | /opt/invenio/bin/dbexec
Not at all: 109, much, much less than the production system.
> Does bibsched eat lots of memory too and is its memory consumption
> rising with time? Or does it only consume CPU?
Only CPU. I haven't seen any memory usage increase.
> Have you tried to run bibsched via profiling?
>
> $ sudo -u www-data python -m cProfile \
> /opt/invenio/lib/python/invenio/bibsched.py
Ah! I didn't know how to profile, great to know. Unfortunately,
bibsched forks itself, and the child apparently doesn't inherit the
profile request, or I don't know where to look at. I see the profiling
result of the front-end monitor process, not the back end.
> Anything interesting in `bibsched.log' and `bibsched.err' or in
> `invenio.err' in `/opt/invenio/var/log/'?
Not really. bibsched.err last update is four days ago, when there was a
pid mismatch. bibshed.log is just a list of started and stopped
processes that look normal.
I've changed all 'debug=False' to 'debug=True' in bibsched.py, but
again, I don't know where to look, being a child process. I'm
monitoring the (internal system) mail and I haven't received anything
yeat.
Mytop gives an obvious clue on the queries sent to the database. I'm
showing both more used. We rebooted the system yesterday after a
scheuled security update, so it is fresh from long-running processes:
MySQL on localhost (5.1.66-0+squeeze1) up 1+08:33:41 [16:57:31]
Queries: 1.1k qps: 0 Slow: 0.0 Se/In/Up/De(%):
45735536/00/00/00
qps now: 0 Slow qps: 0.0 Threads: 17 ( 2/ 6)
534600/00/00/00
Cache Hits: 475.0M Hits/s: 4248.9 Hits now: 2125.9 Ratio: 100.0% Ratio now:
100.0%
Key Efficiency: 100.0% Bps in/out: 0.2/ 38.3 Now in/out: 8.4/ 1.6k
Id User Host/IP DB Time Cmd Query or State
-- ---- ------- -- ---- --- --------------
1723 ddd localhost ddduab 0 Query UPDATE schTASK
SET priority = NULL WHERE status IN ('WAITING', 'RUNNING', 'SLEEPING', 'ABOUT
TO STOP', 'ABOUT TO S
1723 ddd localhost ddduab 0 Query SELECT id, proc,
runtime, status, priority, host, sequenceid FROM schTASK WHERE status IN
('WAITING', 'SLEEPING')
I've verified that CFG_BIBSCHED_REFRESHTIME has the default 5 seconds
value. If I could see the results of the profile!
Thanks,
Ferran
