Hello Jan, I don't see anything useful, if debug logging is disabled in taskomatic.
Also when debugging is disabled, the only thing I see, is, that some errata-cache tasks are "SKIPPED". I also enabled all logging of SQL statements in postgres but - IMHO - everything looks " good". I already started the whole server, restarted taskomatic, restarted spacewalk-service. Nothing seems to help. I'm not sure, if sometimes the threads are colliding, so I wanted to set the number of workers to 1 for some time. On my test system, everything looks good. But there are only about 10 systems registered, so I can't tell, if this "might" be some kind of performance problem. Regards Robert Am 14.01.2016 3:52 nachm. schrieb Jan Dobes <[email protected]>: > > On 12.1.2016 09:08 [email protected] wrote: > > Hello again, > > > > still having problems, > > > > I noticed that the errata-cache gets triggerd when a client refreshes > > its repos or searches for updates. So I manually fired a "zypper ref -s" > > or a "apt-get update" on every client connected to this spacewalk > > server. This seems to have fixed "something". Suddenly, the > > "errata-cache" task has finished and I got some backtraces from spacewalk. > > > > But I still have trouble. The UI does not seem to get updated correctly > > (on the "system" page, nearly all systems look "up-to-date". But I know, > > it's not!). But the updates are available to the clients (searching on > > the client). > > > > When I look into the last errata-cache schedules (in Spacewalk "Admin" > > -> "Task schedules" -> "errata-cache-bunch") I can see, that the > > "errata-cache" tasks is often "SKIPPED" > > > > errata-cache _2016-01-12 08:50:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:50:00 MEZ SKIPPED > > errata-cache _2016-01-12 08:49:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:49:00 MEZ SKIPPED > > errata-cache _2016-01-12 08:48:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:48:00 MEZ SKIPPED > > errata-cache _2016-01-12 08:47:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:50:00 MEZ FINISHED > > errata-cache _2016-01-12 08:46:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:46:00 MEZ SKIPPED > > errata-cache _2016-01-12 08:45:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:45:00 MEZ SKIPPED > > errata-cache _2016-01-12 08:44:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:46:00 MEZ FINISHED > > errata-mailer _2016-01-12 08:43:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:43:00 MEZ FINISHED > > errata-cache _2016-01-12 08:43:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:43:00 MEZ FINISHED > > errata-cache _2016-01-12 08:42:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:42:00 MEZ SKIPPED > > errata-cache _2016-01-12 08:41:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:41:00 MEZ SKIPPED > > errata-cache _2016-01-12 08:40:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:40:00 MEZ SKIPPED > > errata-cache _2016-01-12 08:39:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:42:00 MEZ FINISHED > > errata-mailer _2016-01-12 08:38:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:38:00 MEZ FINISHED > > errata-cache _2016-01-12 08:38:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:38:00 MEZ FINISHED > > errata-mailer _2016-01-12 08:37:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:37:00 MEZ FINISHED > > errata-cache _2016-01-12 08:37:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:37:00 MEZ FINISHED > > errata-cache _2016-01-12 08:36:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:36:00 MEZ SKIPPED > > errata-cache _2016-01-12 08:35:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:35:00 MEZ SKIPPED > > errata-cache _2016-01-12 08:34:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:36:00 MEZ FINISHED > > errata-mailer _2016-01-12 08:33:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:33:00 MEZ FINISHED > > errata-cache _2016-01-12 08:33:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:33:00 MEZ FINISHED > > errata-mailer _2016-01-12 08:32:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:32:00 MEZ FINISHED > > errata-cache _2016-01-12 08:32:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:32:00 MEZ FINISHED > > errata-mailer _2016-01-12 08:31:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:31:00 MEZ FINISHED > > errata-cache _2016-01-12 08:31:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:31:00 MEZ FINISHED > > errata-mailer _2016-01-12 08:30:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:30:00 MEZ FINISHED > > errata-cache _2016-01-12 08:30:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:30:00 MEZ FINISHED > > errata-cache _2016-01-12 08:29:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=132> > > 2016-01-12 08:29:00 MEZ SKIPPED > > > > > > > > And also I noticed that the "errata-mailer" tasks seems to get only > > started, when the "errata-cache" tasks finished within one minute. (Just > > look above). As soon as the errata-cache tasks runs longer than one > > minute, the "errata-mailer" is not started. I'm also not sure, if > > something went wrong, while I was shutting down the spacewalk server for > > the update. > > > > This is what it looked before (on spacewalk 2.2). I started the update > > on 2016-01-04 abount 08:30. > > errata-cache _2016-01-04 10:10:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 10:10:00 MEZ SKIPPED > > errata-cache _2016-01-04 10:00:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 10:00:00 MEZ SKIPPED > > errata-cache _2016-01-04 09:50:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 09:50:00 MEZ SKIPPED > > errata-cache _2016-01-04 09:40:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 09:40:00 MEZ SKIPPED > > errata-cache _2016-01-04 09:30:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 09:30:00 MEZ SKIPPED > > errata-cache _2016-01-04 09:20:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 09:20:00 MEZ SKIPPED > > errata-cache _2016-01-04 09:10:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 09:10:00 MEZ SKIPPED > > errata-cache _2016-01-04 09:00:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 09:00:00 MEZ SKIPPED > > errata-cache _2016-01-04 08:50:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 08:50:00 MEZ SKIPPED > > errata-cache _2016-01-04 08:40:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 08:40:00 MEZ SKIPPED > > errata-cache _2016-01-04 08:30:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-05 08:59:59 MEZ INTERRUPTED > > errata-mailer _2016-01-04 08:30:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 08:29:57 MEZ INTERRUPTED > > errata-mailer _2016-01-04 08:00:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 08:00:00 MEZ FINISHED > > errata-cache _2016-01-04 08:00:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 08:00:00 MEZ FINISHED > > errata-mailer _2016-01-04 07:50:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 07:50:00 MEZ FINISHED > > errata-cache _2016-01-04 07:50:00 MEZ _ > > <https://spacewalk.swr.ard/rhn/admin/ScheduleDetail.do?schid=7> > > 2016-01-04 07:50:00 MEZ FINISHED > > > > > > > > I would be glad, if someone could give me further hints, where to look > > for the root cause to this. > > > > Kind regards, > > Robert > > > > > > Mit freundlichen Grüßen > > > > Robert Paschedag > > Netlution GmbH > > Landteilstr. 33 > > 68163 Mannheim > > > > im Auftrag des > > SWR > > Südwestrundfunk > > Informations- und Kommunikationssysteme > > Neckarstraße 230 > > 70190 Stuttgart > > > > Telefon +49 (0)711 /929-12654 oder > > Telefon +49 (0)711 /929-13714 > > [email protected] > > > > swr.de > > > > > > > > > > > > Von: Robert Paschedag <[email protected]> > > An: [email protected] > > Kopie: [email protected] > > Datum: 08.01.2016 21:53 > > Betreff: Re: [Spacewalk-list] Errata Cache bunch running forever on > > Spacewalk 2.4 > > ------------------------------------------------------------------------ > > > > > > > > Hi again, > > > > just some further information. > > > > Because I already activated debug logging of taskomatic, searching for > > server errata cache tasks within the rhn_taskomatic_daemon logfiles with > > > > grep 'Updating errata cache for server' rhn_taskomatic_daemon.log* | awk > > '{print $NF; }' | sort | uniq -c > > > > i get > > > > 14 [1000010274] > > 14 [1000010299] > > 4 [1000010504] > > 14 [1000010512] > > 14 [1000010516] > > 14 [1000010517] > > 14 [1000010518] > > 14 [1000010519] > > 14 [1000010520] > > 14 [1000010521] > > 14 [1000010522] > > 14 [1000010523] > > 14 [1000010524] > > 14 [1000010527] > > 14 [1000010528] > > 14 [1000010529] > > 14 [1000010530] > > 14 [1000010531] > > 14 [1000010532] > > 14 [1000010564] > > 14 [1000010570] > > 15 [1000010573] > > 15 [1000010593] > > 13 [1000010639] > > 14 [1000010640] > > 15 [1000010641] > > 13 [1000010642] > > 14 [1000010647] > > 13 [1000010648] > > 14 [1000010649] > > 15 [1000010650] > > 14 [1000010651] > > 14 [1000010652] > > 6 [1000010672] > > 15 [1000010683] > > 14 [1000010684] > > 14 [1000010685] > > 14 [1000010687] > > 14 [1000010688] > > 14 [1000010689] > > 14 [1000010690] > > 14 [1000010691] > > 14 [1000010692] > > 14 [1000010693] > > 14 [1000010694] > > 15 [1000010695] > > 14 [1000010697] > > 14 [1000010698] > > 2 [1000010702] > > 14 [1000010705] > > 14 [1000010710] > > 2 [1000010715] > > > > On this server, there are more than 400 servers registered, so this > > looks like, something is "hanging" on these servers and not going any > > further (i think). > > > > But right now, I don't know - exactly - where to look for this "error" > > or within which DB table to further track this down. > > > > Regards, > > Robert > > > > Am 08.01.2016 um 19:31 schrieb Robert Paschedag: > > > Can anybody give a hint, in which db table to search for these > > errata-cache tasks, that are started over and over again? I just don't > > get it myself right now . It also looks like that it is always the same > > prox. 50 servers that this occurs. > > > > > > Regards > > > Robert > > > Am 07.01.2016 10:11 vorm. schrieb [email protected]: > > >> > > >> Hi all, > > >> > > >> it looks like one (or more?) jobs have been interrupted, while the > > spacewalk service had shut down for upgrade. I upgraded my spacewalk > > server from 2.2 -> 2.3 -> 2.4. > > >> > > >> Now it looks like the errata-cache bunch tasks is running forever > > and the errata notification mail is not started anymore. I think it gets > > called from errata-cache, when this has finished. > > >> > > >> Errata Cache: > > >> 2016-01-07 10:00:00 MEZ > > >> RUNNING > > >> Errata Notification Mail: > > >> 2016-01-04 08:30:00 MEZ > > >> INTERRUPTED > > >> > > >> > > >> When I look into the database, I can see several tasks in the > > rhntaskqueue but I'm pretty sure, that these tasks do not get cleared > > correctly, so the task is run for every server over and over again. I'm > > not sure! This is just my thinking. > > >> > > >> rhnschema=# select * from rhntaskqueue; > > >> org_id | task_name | task_data | priority | earliest > > >> > > --------+----------------------------+------------+----------+------------------------------- > > > > > > >> 1 | update_server_errata_cache | 1000010701 | 0 | > > 2016-01-07 10:00:45.963006+01 > > >> 1 | update_server_errata_cache | 1000010274 | 0 | > > 2016-01-07 10:00:53.198001+01 > > >> 1 | update_server_errata_cache | 1000010527 | 0 | > > 2016-01-07 10:01:03.768592+01 > > >> 1 | update_server_errata_cache | 1000010689 | 0 | > > 2016-01-07 10:01:56.915296+01 > > >> 1 | update_server_errata_cache | 1000010641 | 0 | > > 2016-01-07 10:02:41.775593+01 > > >> 1 | update_server_errata_cache | 1000010687 | 0 | > > 2016-01-07 10:05:01.471365+01 > > >> 1 | update_server_errata_cache | 1000010650 | 0 | > > 2016-01-07 10:05:26.492405+01 > > >> 1 | update_server_errata_cache | 1000010520 | 0 | > > 2016-01-07 10:06:38.052352+01 > > >> (8 Zeilen) > > >> > > >> rhnschema=# > > >> > > >> Any help will be appreciated to debug this further. > > >> > > >> Regards, > > >> Robert > > >> > > >> > > >> Mit freundlichen Grüßen > > >> > > >> Robert Paschedag > > >> Netlution GmbH > > >> Landteilstr. 33 > > >> 68163 Mannheim > > >> > > >> im Auftrag des > > >> SWR > > >> Südwestrundfunk > > >> Informations- und Kommunikationssysteme > > >> Neckarstraße 230 > > >> 70190 Stuttgart > > >> > > >> Telefon +49 (0)711 /929-12654 oder > > >> Telefon +49 (0)711 /929-13714 > > >> [email protected] > > >> > > >> swr.de > > >> > > > > > > _______________________________________________ > > > Spacewalk-list mailing list > > > [email protected] > > > https://www.redhat.com/mailman/listinfo/spacewalk-list > > > > > > > > > > > Hello, > > since Spacewalk 2.3 this type of task is running every minute instead of > every ten minutes before. However any of these tasks should not take so > long. These tasks are picked by taskomatic daemon and processed. Are > there any exceptions in /var/log/rhn/rhn_taskomatic_daemon.log ? Try to > restart taskomatic. > > Regards, > -- > Jan Dobes > Satellite Engineering, Red Hat > > _______________________________________________ > Spacewalk-list mailing list > [email protected] > https://www.redhat.com/mailman/listinfo/spacewalk-list _______________________________________________ Spacewalk-list mailing list [email protected] https://www.redhat.com/mailman/listinfo/spacewalk-list
